[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:30 dist-test-slave-jtnz krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:30 dist-test-slave-jtnz krb5kdc[399](info): set up 2 sockets
Sep 24 03:08:30 dist-test-slave-jtnz krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 03:08:31.307060 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:31.412993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:31.489324 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:31.489504 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:31.489526 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:31.490579 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:31.490612 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:31.490633 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:31.490646 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:31.490658 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:31.491994 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:31.492159 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:31.492331 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:31.497053 412 server_base.cc:1034] running on GCE node
W20260924 03:08:31.498627 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.499078 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.499176 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:31.499410 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:31.499643 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:31.500777 412 hybrid_clock.cc:648] HybridClock initialized: now 1790219311500762 us; error 34 us; skew 500 ppm
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:31.503235 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:31.503671 412 webserver.cc:460] Webserver started at http://127.0.98.62:39775/ using document root <none> and password file <none>
I20260924 03:08:31.503832 412 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:31.503868 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:31.503947 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:31.504884 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "fd1248c9c85f4a988f565deb4b83b70c"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "be28fe8e70ace47d79f6f0b67dc11df9"
server_key_iv: "6bbd7c6e34de21f0882b3d07b59cf2fc"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.505226 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd1248c9c85f4a988f565deb4b83b70c"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "be28fe8e70ace47d79f6f0b67dc11df9"
server_key_iv: "6bbd7c6e34de21f0882b3d07b59cf2fc"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.506474 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:31.507097 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.507279 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:31.507364 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "fd1248c9c85f4a988f565deb4b83b70c"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "be28fe8e70ace47d79f6f0b67dc11df9"
server_key_iv: "6bbd7c6e34de21f0882b3d07b59cf2fc"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.507447 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:31.523069 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:31.525388 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:31.525556 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:31.529289 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45145
I20260924 03:08:31.529317 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45145 every 8 connection(s)
I20260924 03:08:31.529635 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:31.530496 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:31.532493 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c: Bootstrap starting.
I20260924 03:08:31.533033 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:31.533305 480 log.cc:824] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:31.533967 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c: No bootstrap required, opened a new log
I20260924 03:08:31.535475 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260924 03:08:31.535605 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:31.535648 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1248c9c85f4a988f565deb4b83b70c, State: Initialized, Role: FOLLOWER
I20260924 03:08:31.535807 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260924 03:08:31.535868 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:31.535908 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:31.535957 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:31.536509 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260924 03:08:31.536628 480 leader_election.cc:302] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1248c9c85f4a988f565deb4b83b70c; no voters:
I20260924 03:08:31.536626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 412
I20260924 03:08:31.536728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:31.536789 480 leader_election.cc:288] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:31.536819 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:31.536991 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [term 1 LEADER]: Becoming Leader. State: Replica: fd1248c9c85f4a988f565deb4b83b70c, State: Running, Role: LEADER
I20260924 03:08:31.537015 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:31.537125 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } }
I20260924 03:08:31.537526 392 external_mini_cluster.cc:1428] Setting key 9402d4a45a86ce5753dcda9c57eb37d3
I20260924 03:08:31.537568 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1248c9c85f4a988f565deb4b83b70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } } }
I20260924 03:08:31.537658 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:31.538062 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:31.538589 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:31.538594 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1248c9c85f4a988f565deb4b83b70c. Latest consensus state: current_term: 1 leader_uuid: "fd1248c9c85f4a988f565deb4b83b70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1248c9c85f4a988f565deb4b83b70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45145 } } }
I20260924 03:08:31.538913 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:31.541249 492 catalog_manager.cc:1347] Generated new cluster ID: 781eb01fead8437da5026b6f0e1e8593
I20260924 03:08:31.541296 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:31.558518 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:31.538814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.98.62:45145)
0924 03:08:31.538980 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:31.538989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:31.539285 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:08:31.540982 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:31.540989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:31.541042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:31.542263 (+ 1221us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:31.543233 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.544604 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.546176 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.546795 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.546890 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:31.554292 (+ 7402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:31.554297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:31.554297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:31.554307 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:31.556237 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.556674 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.556677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.556678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.556777 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.557032 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.557032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.557033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.557451 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:31.557604 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:31.558003 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 03:08:31.558139 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 03:08:31.565940 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:31.566648 492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:31.572496 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1248c9c85f4a988f565deb4b83b70c: Generated new TSK 0
I20260924 03:08:31.572659 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:31.610644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:31.685719 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:31.685900 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:31.685923 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:31.687069 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:31.687126 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:31.687155 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:31.688539 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:31.688750 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:31.688944 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:31.693446 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.693459 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.693459 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:31.693621 507 server_base.cc:1034] running on GCE node
I20260924 03:08:31.693858 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:31.694084 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:31.695235 507 hybrid_clock.cc:648] HybridClock initialized: now 1790219311695228 us; error 38 us; skew 500 ppm
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:31.697535 507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:31.698001 507 webserver.cc:460] Webserver started at http://127.0.98.1:45395/ using document root <none> and password file <none>
I20260924 03:08:31.698166 507 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:31.698256 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:31.698354 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:31.699237 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "08d7f08f8cf64457a251ee730491c911"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "97e15b1b6e1c1ee3fcff49bbb9e05868"
server_key_iv: "452ac4902ab66c0b991b84493c924d88"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.699512 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08d7f08f8cf64457a251ee730491c911"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "97e15b1b6e1c1ee3fcff49bbb9e05868"
server_key_iv: "452ac4902ab66c0b991b84493c924d88"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.700609 507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:08:31.701190 522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.701324 507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.701406 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "08d7f08f8cf64457a251ee730491c911"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "97e15b1b6e1c1ee3fcff49bbb9e05868"
server_key_iv: "452ac4902ab66c0b991b84493c924d88"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.701453 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:31.724237 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:31.725975 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:31.726084 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:31.726310 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:31.726548 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:31.726577 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.726599 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:31.726611 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.731781 507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39149
I20260924 03:08:31.731856 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39149 every 8 connection(s)
I20260924 03:08:31.732167 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:31.735066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 507
I20260924 03:08:31.735165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:31.735298 392 external_mini_cluster.cc:1428] Setting key bdcb7131443634c9d6d5639193ca7242
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:31.745154 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:31.732874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33515 (local address 127.0.98.62:45145)
0924 03:08:31.732916 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:31.732919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:31.733096 (+ 177us) server_negotiation.cc:407] Connection header received
0924 03:08:31.733579 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:31.733580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:31.733596 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:31.733649 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:31.734555 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.735045 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.736209 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.736492 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.736572 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:31.742623 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:31.742625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:31.742625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:31.742633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:31.743940 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.744378 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.744379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.744379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.744427 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.744616 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.744617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.744617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.744712 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:31.744761 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:31.745030 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:08:31.745094 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 03:08:31.745435 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:31.732833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.1:33515)
0924 03:08:31.732972 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:31.733014 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:31.733479 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:31.733729 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:31.733733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:31.734278 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:31.734451 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:31.734456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.735133 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.735134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:31.736122 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:31.736124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.736580 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.736582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:31.736715 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:31.737230 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:31.737241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:31.742505 (+ 5264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:31.744106 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:31.744111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:31.744113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:31.744317 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:31.744480 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:31.744480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:31.744481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:31.744566 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:31.744769 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:31.744775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:31.744920 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 03:08:31.745005 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:08:31.745114 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260924 03:08:31.745959 636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260924 03:08:31.746054 636 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:31.746256 636 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260924 03:08:31.746826 444 ts_manager.cc:194] Registered new tserver with Master: 08d7f08f8cf64457a251ee730491c911 (127.0.98.1:39149)
I20260924 03:08:31.747726 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:31.759135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:31.832216 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:31.832381 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:31.832398 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:31.833453 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:31.833491 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:31.833539 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:31.834894 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:31.835107 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:31.835309 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:31.839963 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.839967 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:31.840041 641 server_base.cc:1034] running on GCE node
W20260924 03:08:31.840171 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:31.840368 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:31.840530 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:31.841667 641 hybrid_clock.cc:648] HybridClock initialized: now 1790219311841657 us; error 25 us; skew 500 ppm
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:31.843957 641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:31.844521 641 webserver.cc:460] Webserver started at http://127.0.98.2:38431/ using document root <none> and password file <none>
I20260924 03:08:31.844725 641 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:31.844766 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:31.844907 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:31.845817 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "96f01378de4e4823bcb041a2e1d6e5a7"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "c6cd6f970eed5642f1b8659af9c9a6af"
server_key_iv: "3b18f30ffb23bad72dd9a8e90160b543"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.846223 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96f01378de4e4823bcb041a2e1d6e5a7"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "c6cd6f970eed5642f1b8659af9c9a6af"
server_key_iv: "3b18f30ffb23bad72dd9a8e90160b543"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.847451 641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:08:31.848153 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.848310 641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:31.848382 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "96f01378de4e4823bcb041a2e1d6e5a7"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "c6cd6f970eed5642f1b8659af9c9a6af"
server_key_iv: "3b18f30ffb23bad72dd9a8e90160b543"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.848457 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:31.870402 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:31.872066 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:31.872215 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:31.872406 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:31.872681 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:31.872732 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.872771 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:31.872798 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.878599 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32979
I20260924 03:08:31.878690 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32979 every 8 connection(s)
I20260924 03:08:31.878943 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:31.883232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 641
I20260924 03:08:31.883325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:31.883474 392 external_mini_cluster.cc:1428] Setting key ece745bd24c77c68db924fb0d3e38c85
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:31.887508 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:31.879570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.2:58649)
0924 03:08:31.879721 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:31.879752 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:31.880140 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:31.880319 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:31.880322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:31.880801 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:31.880960 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:31.880964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.881595 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.881595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:31.882418 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:31.882421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.882726 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.882726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:31.882809 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:31.883188 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:31.883198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:31.884614 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:31.886147 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:31.886153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:31.886155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:31.886472 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:31.886707 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:31.886707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:31.886708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:31.886798 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:31.887065 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:31.887069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:31.887168 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 03:08:31.887244 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:08:31.887357 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260924 03:08:31.887542 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:31.879611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58649 (local address 127.0.98.62:45145)
0924 03:08:31.879692 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:31.879694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:31.879812 (+ 118us) server_negotiation.cc:407] Connection header received
0924 03:08:31.880217 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:31.880218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:31.880227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:31.880275 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:31.881042 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.881521 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.882485 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:31.882660 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:31.882711 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:31.884700 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:31.884701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:31.884702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:31.884708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:31.886046 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.886574 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.886575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.886575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.886640 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:31.886861 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:31.886862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:31.886863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:31.887004 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:31.887071 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:31.887288 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 03:08:31.887387 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:08:31.888099 770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260924 03:08:31.888196 770 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:31.888376 770 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260924 03:08:31.888733 444 ts_manager.cc:194] Registered new tserver with Master: 96f01378de4e4823bcb041a2e1d6e5a7 (127.0.98.2:32979)
I20260924 03:08:31.889084 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58649
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:31.906283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:31.976985 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:31.977155 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:31.977178 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:31.978194 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:31.978248 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:31.978288 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:31.979631 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:31.979857 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:31.980070 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:31.984539 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.984577 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:31.984539 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:31.984637 775 server_base.cc:1034] running on GCE node
I20260924 03:08:31.984874 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:31.985052 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:31.986236 775 hybrid_clock.cc:648] HybridClock initialized: now 1790219311986219 us; error 32 us; skew 500 ppm
Sep 24 03:08:31 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:31.988631 775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:31.989053 775 webserver.cc:460] Webserver started at http://127.0.98.3:44867/ using document root <none> and password file <none>
I20260924 03:08:31.989270 775 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:31.989312 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:31.989470 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:31.990428 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec0eaf9cab3e4d3c9e115abda0357c25"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "74af493cc5fd6d8cec15f14a50f82115"
server_key_iv: "11784dc93a7d5f0052a82febd4315265"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.990727 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec0eaf9cab3e4d3c9e115abda0357c25"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "74af493cc5fd6d8cec15f14a50f82115"
server_key_iv: "11784dc93a7d5f0052a82febd4315265"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.991915 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:31.992558 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:31.992712 775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:31.992805 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "ec0eaf9cab3e4d3c9e115abda0357c25"
format_stamp: "Formatted at 2026-09-24 03:08:31 on dist-test-slave-jtnz"
server_key: "74af493cc5fd6d8cec15f14a50f82115"
server_key_iv: "11784dc93a7d5f0052a82febd4315265"
server_key_version: "encryptionkey@0"
I20260924 03:08:31.992873 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:32.007213 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:32.009202 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:32.009336 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:32.009545 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:32.009807 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:32.009855 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:32.009889 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:32.009913 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:32.015209 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34655
I20260924 03:08:32.015265 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34655 every 8 connection(s)
I20260924 03:08:32.015911 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:32.019871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 775
I20260924 03:08:32.019994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:32.020159 392 external_mini_cluster.cc:1428] Setting key 5e856316efd747a6c63fdb607ad20b3f
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:32.024154 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:32.016535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54293 (local address 127.0.98.62:45145)
0924 03:08:32.016600 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:32.016602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:32.016781 (+ 179us) server_negotiation.cc:407] Connection header received
0924 03:08:32.017168 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:32.017169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:32.017178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:32.017226 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:32.017964 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.018475 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.019353 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.019524 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.019573 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:32.021841 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:32.021842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:32.021842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:32.021848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:32.023234 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.023512 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.023512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.023513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.023552 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.023723 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.023723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.023723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.023809 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:32.023850 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:32.024040 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 03:08:32.024101 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:08:32.024323 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:32.016501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45145 (local address 127.0.98.3:54293)
0924 03:08:32.016648 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:32.016697 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:32.017090 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:32.017268 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:32.017271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:32.017744 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:32.017897 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:32.017901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.018540 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.018540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:32.019286 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:32.019287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.019589 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.019589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:32.019696 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:32.020243 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:32.020252 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:32.021768 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:32.023289 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:32.023292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:32.023294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:32.023458 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:32.023598 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:32.023599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:32.023599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:32.023678 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:32.023853 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:32.023856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:32.023960 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 03:08:32.024029 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:08:32.024125 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260924 03:08:32.024775 904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45145
I20260924 03:08:32.024837 904 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:32.024986 904 heartbeater.cc:511] Master 127.0.98.62:45145 requested a full tablet report, sending...
I20260924 03:08:32.025269 444 ts_manager.cc:194] Registered new tserver with Master: ec0eaf9cab3e4d3c9e115abda0357c25 (127.0.98.3:34655)
I20260924 03:08:32.025575 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54293
I20260924 03:08:32.031762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:32.053534 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:32.046698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.98.1:39149)
0924 03:08:32.046868 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:32.046873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:32.046882 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:32.046905 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:32.046907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:32.046918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:32.046981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:32.047483 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.048162 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.048802 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.049089 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.049139 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:32.051234 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:32.051239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:32.051239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:32.051245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:32.052673 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.052939 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.052941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.052942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.052997 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.053146 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.053146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.053146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.053260 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:32.053364 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:32.053414 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 03:08:32.053467 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260924 03:08:32.053911 550 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60986
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:32.072664 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:32.067116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.98.1:39149)
0924 03:08:32.067194 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:32.067195 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:32.067271 (+ 76us) server_negotiation.cc:407] Connection header received
0924 03:08:32.067313 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:32.067314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:32.067323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:32.067369 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:32.067802 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.068246 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.068799 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:32.068953 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:32.068993 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:32.070762 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:32.070763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:32.070764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:32.070770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:32.071833 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.072072 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.072072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.072072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.072112 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:32.072234 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:32.072234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:32.072234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:32.072294 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:32.072361 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:32.072489 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:08:32.072559 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:08:32.073125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 507
I20260924 03:08:32.077819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 641
I20260924 03:08:32.083237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 775
I20260924 03:08:32.087086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 412
2026-09-24T03:08:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1832 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[918](info): set up 2 sockets
Sep 24 03:08:32 dist-test-slave-jtnz krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260924 03:08:33.049353 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:33.151146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:33.224259 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:33.224443 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:33.224463 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:33.225485 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:33.225517 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:33.225530 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:33.225541 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:33.225553 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:33.226833 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:33.227010 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:33.227162 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:33.231498 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:33.231578 931 server_base.cc:1034] running on GCE node
W20260924 03:08:33.231503 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.231504 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:33.231982 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:33.232257 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:33.233409 931 hybrid_clock.cc:648] HybridClock initialized: now 1790219313233398 us; error 24 us; skew 500 ppm
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:33.235616 931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:33.236093 931 webserver.cc:460] Webserver started at http://127.0.98.62:43961/ using document root <none> and password file <none>
I20260924 03:08:33.236299 931 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:33.236340 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:33.236466 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:33.237402 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "fc42829f454c482782f16a6a3bbea49f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "96154c68854b8acbaaf5cbba0c53ddfc"
server_key_iv: "284c2f43ac9663936cb4184487a189a9"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.237733 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc42829f454c482782f16a6a3bbea49f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "96154c68854b8acbaaf5cbba0c53ddfc"
server_key_iv: "284c2f43ac9663936cb4184487a189a9"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.239027 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:33.239692 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.239858 931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:33.239936 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "fc42829f454c482782f16a6a3bbea49f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "96154c68854b8acbaaf5cbba0c53ddfc"
server_key_iv: "284c2f43ac9663936cb4184487a189a9"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.240000 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:33.251430 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:33.252946 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:33.253080 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:33.256872 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41421
I20260924 03:08:33.256923 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41421 every 8 connection(s)
I20260924 03:08:33.257244 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:33.257977 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:33.259939 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f: Bootstrap starting.
I20260924 03:08:33.260458 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:33.260733 999 log.cc:824] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:33.261430 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f: No bootstrap required, opened a new log
I20260924 03:08:33.263051 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260924 03:08:33.263171 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:33.263193 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc42829f454c482782f16a6a3bbea49f, State: Initialized, Role: FOLLOWER
I20260924 03:08:33.263365 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260924 03:08:33.263422 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:33.263456 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:33.263507 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:33.263993 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260924 03:08:33.264096 999 leader_election.cc:302] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc42829f454c482782f16a6a3bbea49f; no voters:
I20260924 03:08:33.264279 999 leader_election.cc:288] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:33.264320 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:33.264518 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:33.264652 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [term 1 LEADER]: Becoming Leader. State: Replica: fc42829f454c482782f16a6a3bbea49f, State: Running, Role: LEADER
I20260924 03:08:33.264740 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260924 03:08:33.264874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 931
I20260924 03:08:33.264956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:33.265117 392 external_mini_cluster.cc:1428] Setting key bc3f6642af61a0e180dfe1902679f7d6
I20260924 03:08:33.265311 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc42829f454c482782f16a6a3bbea49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260924 03:08:33.265385 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:33.265520 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc42829f454c482782f16a6a3bbea49f. Latest consensus state: current_term: 1 leader_uuid: "fc42829f454c482782f16a6a3bbea49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc42829f454c482782f16a6a3bbea49f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260924 03:08:33.265604 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:33.265710 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:33.266129 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:33.267509 1021 mvcc.cc:205] Tried to move back new op lower bound from 7332738307141238784 to 7332738307132788736. Current Snapshot: MvccSnapshot[applied={T|T < 7332738307141238784}]
I20260924 03:08:33.267870 1011 catalog_manager.cc:1347] Generated new cluster ID: 5f3dbd8e9cc24eb389b565d6edbe2717
I20260924 03:08:33.267920 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:33.277297 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.265736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.98.62:41421)
0924 03:08:33.265875 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.265883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.266850 (+ 967us) server_negotiation.cc:407] Connection header received
0924 03:08:33.267003 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.267010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.267071 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.267204 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.268132 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.269489 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.270574 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.271168 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.271261 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.273911 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.273915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.273915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.273926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.275776 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.276127 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.276129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.276130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.276227 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.276412 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.276412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.276413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.276703 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.276847 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.276924 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.277079 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260924 03:08:33.281666 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:33.282389 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:33.290321 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc42829f454c482782f16a6a3bbea49f: Generated new TSK 0
I20260924 03:08:33.290481 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:33.318301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:33.395959 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:33.396178 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:33.396211 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:33.397334 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:33.397400 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:33.397444 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:33.398831 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:33.399083 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:33.399308 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:33.403935 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.403947 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.403947 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:33.404181 1025 server_base.cc:1034] running on GCE node
I20260924 03:08:33.404361 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:33.404569 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:33.405704 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790219313405689 us; error 31 us; skew 500 ppm
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:33.408097 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:33.408553 1025 webserver.cc:460] Webserver started at http://127.0.98.1:43173/ using document root <none> and password file <none>
I20260924 03:08:33.408756 1025 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:33.408797 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:33.408936 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:33.409878 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "49d10cbc7f774a8bb224a5e89d492941"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "de276bc89843bf428276c0ba208f41bb"
server_key_iv: "27baa155a86a00227efb9d112fd33d77"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.410282 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49d10cbc7f774a8bb224a5e89d492941"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "de276bc89843bf428276c0ba208f41bb"
server_key_iv: "27baa155a86a00227efb9d112fd33d77"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.411517 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:08:33.412228 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.412408 1025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:33.412499 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "49d10cbc7f774a8bb224a5e89d492941"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "de276bc89843bf428276c0ba208f41bb"
server_key_iv: "27baa155a86a00227efb9d112fd33d77"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.412582 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:33.429911 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:33.431671 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:33.431818 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:33.432019 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:33.432343 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:33.432384 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.432420 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:33.432446 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.438040 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36441
I20260924 03:08:33.438076 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36441 every 8 connection(s)
I20260924 03:08:33.438395 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:33.442577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1025
I20260924 03:08:33.442675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:33.442834 392 external_mini_cluster.cc:1428] Setting key f40d41e2b2699568a85cea900aa56b91
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:33.447817 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.439092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47015 (local address 127.0.98.62:41421)
0924 03:08:33.439150 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.439152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.439352 (+ 200us) server_negotiation.cc:407] Connection header received
0924 03:08:33.439873 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.439875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.439886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.439948 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.440825 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.441322 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.442403 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.442613 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.442697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.445086 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.445088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.445088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.445098 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.446473 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.446845 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.446846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.446846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.446904 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.447184 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.447185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.447185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.447299 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.447388 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.447650 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.447726 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:08:33.448010 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.439050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:47015)
0924 03:08:33.439242 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.439274 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.439770 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.440021 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.440025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.440553 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:33.440724 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.440729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.441414 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.441415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.442283 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.442286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.442705 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.442706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.442858 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.443365 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:33.443375 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:33.444965 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:33.446549 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.446555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.446557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.446772 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.446989 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.446990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.446991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.447098 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.447381 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:33.447387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:33.447518 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.447629 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.447777 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260924 03:08:33.448452 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260924 03:08:33.448558 1154 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:33.448743 1154 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260924 03:08:33.449216 963 ts_manager.cc:194] Registered new tserver with Master: 49d10cbc7f774a8bb224a5e89d492941 (127.0.98.1:36441)
I20260924 03:08:33.449813 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47015
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:33.466434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:33.546337 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:33.546510 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:33.546532 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:33.547595 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:33.547628 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:33.547653 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:33.548933 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:33.549086 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:33.549230 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:33.553649 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.553692 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.553712 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:33.553750 1159 server_base.cc:1034] running on GCE node
I20260924 03:08:33.553934 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:33.554220 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:33.555361 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790219313555346 us; error 30 us; skew 500 ppm
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:33.557785 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:33.558336 1159 webserver.cc:460] Webserver started at http://127.0.98.2:46421/ using document root <none> and password file <none>
I20260924 03:08:33.558507 1159 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:33.558545 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:33.558624 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:33.559507 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "809b1c85ffc541cc911218e35527978f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "1ea66194f0a0e3cffabc65044656908a"
server_key_iv: "cfd55e5066f9f961ed727299fe40ef80"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.559829 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "809b1c85ffc541cc911218e35527978f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "1ea66194f0a0e3cffabc65044656908a"
server_key_iv: "cfd55e5066f9f961ed727299fe40ef80"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.561100 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:08:33.561894 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.562093 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:33.562170 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "809b1c85ffc541cc911218e35527978f"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "1ea66194f0a0e3cffabc65044656908a"
server_key_iv: "cfd55e5066f9f961ed727299fe40ef80"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.562276 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:33.594403 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:33.596216 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:33.596359 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:33.596570 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:33.596853 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:33.596884 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.596903 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:33.596951 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.602583 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35587
I20260924 03:08:33.602640 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35587 every 8 connection(s)
I20260924 03:08:33.602944 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:33.610509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1159
I20260924 03:08:33.610576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:33.610718 392 external_mini_cluster.cc:1428] Setting key 348c4bbeda8ac9e5d0964f2e6c7cbaa0
I20260924 03:08:33.611750 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.603735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40169 (local address 127.0.98.62:41421)
0924 03:08:33.603805 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.603807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.604006 (+ 199us) server_negotiation.cc:407] Connection header received
0924 03:08:33.604467 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.604468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.604476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.604525 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.605340 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.605803 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.606736 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.606922 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.607014 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.608955 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.608957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.608957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.608966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.610313 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.610748 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.610749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.610749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.610839 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.611141 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.611142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.611142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.611268 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.611341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.611636 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.611693 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:08:33.612015 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.603724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:40169)
0924 03:08:33.603885 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.603923 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.604395 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.604571 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.604575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.605097 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:33.605271 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.605274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.605898 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.605899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.606649 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.606651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.607031 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.607032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.607148 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.607597 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:33.607604 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:33.608856 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:33.610413 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.610418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.610420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.610675 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.610971 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.610971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.610971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.611058 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.611354 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:33.611360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:33.611504 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.611618 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.611770 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260924 03:08:33.612579 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260924 03:08:33.612682 1288 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:33.612836 1288 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260924 03:08:33.613183 963 ts_manager.cc:194] Registered new tserver with Master: 809b1c85ffc541cc911218e35527978f (127.0.98.2:35587)
I20260924 03:08:33.613683 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40169
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:33.634516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:33.708832 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:33.709028 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:33.709060 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:33.710177 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:33.710250 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:33.710291 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:33.711592 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:33.711869 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:33.712073 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:33.716612 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.716634 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:33.716615 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:33.716647 1293 server_base.cc:1034] running on GCE node
I20260924 03:08:33.716970 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:33.717199 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:33.718345 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790219313718331 us; error 26 us; skew 500 ppm
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:33.720515 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:33.720873 1293 webserver.cc:460] Webserver started at http://127.0.98.3:38575/ using document root <none> and password file <none>
I20260924 03:08:33.721072 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:33.721129 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:33.721247 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:33.722124 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "86646fecc8914e27b6860fe81946a1ae"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "7c6572501030ca47474bb34d818d6e27"
server_key_iv: "f33413f5d5c3849a497a066cedf45e64"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.722514 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "86646fecc8914e27b6860fe81946a1ae"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "7c6572501030ca47474bb34d818d6e27"
server_key_iv: "f33413f5d5c3849a497a066cedf45e64"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.723707 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:33.724359 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.724548 1293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:33.724639 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "86646fecc8914e27b6860fe81946a1ae"
format_stamp: "Formatted at 2026-09-24 03:08:33 on dist-test-slave-jtnz"
server_key: "7c6572501030ca47474bb34d818d6e27"
server_key_iv: "f33413f5d5c3849a497a066cedf45e64"
server_key_version: "encryptionkey@0"
I20260924 03:08:33.724720 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:33.733958 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:33.735536 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:33.735675 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:33.735855 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:33.736106 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:33.736155 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.736191 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:33.736214 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:33.741287 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46807
I20260924 03:08:33.741344 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46807 every 8 connection(s)
I20260924 03:08:33.741612 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:33.748183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1293
I20260924 03:08:33.748275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:33.748432 392 external_mini_cluster.cc:1428] Setting key 564f587a3a1ae06d6d619967aba7440d
I20260924 03:08:33.749950 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.742253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37767 (local address 127.0.98.62:41421)
0924 03:08:33.742303 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.742305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.742466 (+ 161us) server_negotiation.cc:407] Connection header received
0924 03:08:33.742885 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.742886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.742896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.742945 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.743805 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.744274 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.745196 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.745368 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.745417 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.747285 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.747286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.747286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.747292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.748704 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.749115 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.749116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.749117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.749167 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.749397 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.749398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.749398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.749500 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.749562 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.749803 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.749870 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:08:33.750115 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.742248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:37767)
0924 03:08:33.742380 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.742408 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.742812 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.742990 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.742993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.743542 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:33.743741 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.743744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.744347 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.744347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.745118 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.745119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.745435 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.745436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.745527 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.745910 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:33.745917 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:33.747170 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:33.748778 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.748783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.748785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.749041 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.749234 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:33.749235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:33.749235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:33.749333 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:33.749564 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:33.749570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:33.749690 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.749781 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.749909 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 03:08:33.750552 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260924 03:08:33.750638 1422 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:33.750782 1422 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260924 03:08:33.751094 963 ts_manager.cc:194] Registered new tserver with Master: 86646fecc8914e27b6860fe81946a1ae (127.0.98.3:46807)
I20260924 03:08:33.751456 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37767
I20260924 03:08:33.760021 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:33.767084 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.761420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.98.62:41421)
0924 03:08:33.761487 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.761490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.761536 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:08:33.761580 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.761581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.761588 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.761624 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.762289 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.762767 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.763427 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.763593 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.763645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.764729 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.764730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.764730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.764742 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.766081 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.766437 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.766438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.766438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.766488 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.766660 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.766660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.766661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.766764 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.766831 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.766913 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.766970 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:08:33.768899 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:33.770385 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:33.776168 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35587 (local address 127.0.0.1:40162)
0924 03:08:33.772950 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.772961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.773005 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.773201 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.773202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.773204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:33.773235 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.773236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.774429 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.774430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.775160 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.775162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.775899 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.775900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.775973 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.775985 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.776053 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.776097 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":29,"threads_started":1}
I20260924 03:08:33.776176 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.0.98.2:35587)
0924 03:08:33.772909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.772913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.773054 (+ 141us) server_negotiation.cc:407] Connection header received
0924 03:08:33.773092 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.773094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.773117 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.773190 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:33.773342 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.774335 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.775254 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.775800 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.775899 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.775953 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.776065 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.776101 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260924 03:08:33.776369 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36441 (local address 127.0.0.1:41120)
0924 03:08:33.772367 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.772400 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.772458 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.772829 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.772832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.772834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:33.772891 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.772893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.773956 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.773958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.775221 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.775224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.776121 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.776122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.776195 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.776198 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.776246 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.776289 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20260924 03:08:33.776369 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46807 (local address 127.0.0.1:38914)
0924 03:08:33.772515 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.772525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.772574 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.772900 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.772901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.772903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:33.772953 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.772955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.774120 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.774121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.775227 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.775228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.776156 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.776157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.776209 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.776212 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.776276 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.776304 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":51,"threads_started":1}
I20260924 03:08:33.776527 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.3:46807)
0924 03:08:33.772679 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.772683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.772709 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:08:33.772741 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.772743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.772756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.772836 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:33.773016 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.774029 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.775320 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.776014 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.776080 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.776130 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.776419 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.776461 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 03:08:33.776695 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.772367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.98.1:36441)
0924 03:08:33.772619 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.772623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.772633 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:08:33.772672 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.772674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.772710 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.772809 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:33.772970 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.773863 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.775320 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.776014 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.776499 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.776558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.776582 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.776624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":69,"threads_started":1}
I20260924 03:08:33.777161 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 6f263a63f0344ca9a7939e1aca32cbe4 (DEFAULT_TABLE table=test-table [id=9195458b74da41aba819900b9c58e8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:33.777417 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 6f263a63f0344ca9a7939e1aca32cbe4 (DEFAULT_TABLE table=test-table [id=9195458b74da41aba819900b9c58e8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:33.777545 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f263a63f0344ca9a7939e1aca32cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:33.777532 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 6f263a63f0344ca9a7939e1aca32cbe4 (DEFAULT_TABLE table=test-table [id=9195458b74da41aba819900b9c58e8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:33.777740 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f263a63f0344ca9a7939e1aca32cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:33.777801 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f263a63f0344ca9a7939e1aca32cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:33.779898 1436 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: Bootstrap starting.
I20260924 03:08:33.780999 1436 tablet_bootstrap.cc:654] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:33.781051 1438 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Bootstrap starting.
I20260924 03:08:33.781472 1436 log.cc:824] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:33.781605 1438 tablet_bootstrap.cc:654] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:33.782110 1438 log.cc:824] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:33.782184 1436 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: No bootstrap required, opened a new log
I20260924 03:08:33.782282 1436 ts_tablet_manager.cc:1397] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:08:33.782428 1437 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Bootstrap starting.
I20260924 03:08:33.782894 1437 tablet_bootstrap.cc:654] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:33.783316 1437 log.cc:824] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:33.783742 1438 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: No bootstrap required, opened a new log
I20260924 03:08:33.783659 1436 raft_consensus.cc:348] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.783814 1438 ts_tablet_manager.cc:1397] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:08:33.783845 1436 raft_consensus.cc:374] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:33.783890 1436 raft_consensus.cc:729] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86646fecc8914e27b6860fe81946a1ae, State: Initialized, Role: FOLLOWER
I20260924 03:08:33.783998 1436 consensus_queue.cc:260] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.784144 1437 tablet_bootstrap.cc:492] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: No bootstrap required, opened a new log
I20260924 03:08:33.784216 1437 ts_tablet_manager.cc:1397] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:33.784265 1436 ts_tablet_manager.cc:1428] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:08:33.784386 1422 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260924 03:08:33.785629 1438 raft_consensus.cc:348] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.785799 1438 raft_consensus.cc:374] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:33.785831 1438 raft_consensus.cc:729] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d10cbc7f774a8bb224a5e89d492941, State: Initialized, Role: FOLLOWER
I20260924 03:08:33.785928 1438 consensus_queue.cc:260] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.785986 1437 raft_consensus.cc:348] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.786149 1437 raft_consensus.cc:374] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:33.786170 1438 ts_tablet_manager.cc:1428] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:08:33.786209 1437 raft_consensus.cc:729] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809b1c85ffc541cc911218e35527978f, State: Initialized, Role: FOLLOWER
I20260924 03:08:33.786206 1154 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260924 03:08:33.786312 1437 consensus_queue.cc:260] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.786583 1437 ts_tablet_manager.cc:1428] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:08:33.786588 1288 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
W20260924 03:08:33.853564 1289 tablet.cc:2369] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:33.903561 1442 raft_consensus.cc:482] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:33.903697 1442 raft_consensus.cc:504] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.904034 1442 leader_election.cc:288] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d10cbc7f774a8bb224a5e89d492941 (127.0.98.1:36441), 809b1c85ffc541cc911218e35527978f (127.0.98.2:35587)
I20260924 03:08:33.907972 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.904306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43927 (local address 127.0.98.1:36441)
0924 03:08:33.904351 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.904354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.904365 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:33.904395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.904396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.904408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.904460 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:33.904622 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.905339 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907040 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.907636 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907707 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.907716 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.907870 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.907905 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:08:33.907972 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.904393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57119 (local address 127.0.98.2:35587)
0924 03:08:33.904431 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.904433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.904547 (+ 114us) server_negotiation.cc:407] Connection header received
0924 03:08:33.904609 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.904611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.904623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.904676 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:33.904803 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.905454 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907039 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.907632 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907707 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.907716 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.907870 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.907908 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:08:33.907979 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.904212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36441 (local address 127.0.98.3:43927)
0924 03:08:33.904276 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.904287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.904344 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.904476 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.904477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.904479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:33.904551 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.904553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.905396 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.905396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.906927 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.906930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.907718 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.907794 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.907796 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.907849 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.907898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":43}
I20260924 03:08:33.907986 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.904324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35587 (local address 127.0.98.3:57119)
0924 03:08:33.904484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:33.904492 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:33.904541 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:33.904697 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:33.904697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:33.904698 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:33.904742 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.904743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.905514 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.905514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.906947 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:33.906949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.907718 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.907719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:33.907791 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.907795 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:33.907849 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 03:08:33.907898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260924 03:08:33.908401 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f263a63f0344ca9a7939e1aca32cbe4" candidate_uuid: "86646fecc8914e27b6860fe81946a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d10cbc7f774a8bb224a5e89d492941" is_pre_election: true
I20260924 03:08:33.908558 1108 raft_consensus.cc:2454] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86646fecc8914e27b6860fe81946a1ae in term 0.
I20260924 03:08:33.908550 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f263a63f0344ca9a7939e1aca32cbe4" candidate_uuid: "86646fecc8914e27b6860fe81946a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809b1c85ffc541cc911218e35527978f" is_pre_election: true
I20260924 03:08:33.908686 1242 raft_consensus.cc:2454] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86646fecc8914e27b6860fe81946a1ae in term 0.
I20260924 03:08:33.908790 1311 leader_election.cc:302] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d10cbc7f774a8bb224a5e89d492941, 86646fecc8914e27b6860fe81946a1ae; no voters:
I20260924 03:08:33.908921 1442 raft_consensus.cc:2781] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:33.908991 1442 raft_consensus.cc:482] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:33.909013 1442 raft_consensus.cc:3037] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:33.909652 1442 raft_consensus.cc:504] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.909770 1442 leader_election.cc:288] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [CANDIDATE]: Term 1 election: Requested vote from peers 49d10cbc7f774a8bb224a5e89d492941 (127.0.98.1:36441), 809b1c85ffc541cc911218e35527978f (127.0.98.2:35587)
I20260924 03:08:33.909920 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f263a63f0344ca9a7939e1aca32cbe4" candidate_uuid: "86646fecc8914e27b6860fe81946a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d10cbc7f774a8bb224a5e89d492941"
I20260924 03:08:33.909945 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f263a63f0344ca9a7939e1aca32cbe4" candidate_uuid: "86646fecc8914e27b6860fe81946a1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809b1c85ffc541cc911218e35527978f"
I20260924 03:08:33.909977 1108 raft_consensus.cc:3037] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:33.910017 1242 raft_consensus.cc:3037] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:33.910744 1242 raft_consensus.cc:2454] T 6f263a63f0344ca9a7939e1aca32cbe4 P 809b1c85ffc541cc911218e35527978f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86646fecc8914e27b6860fe81946a1ae in term 1.
I20260924 03:08:33.910750 1108 raft_consensus.cc:2454] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86646fecc8914e27b6860fe81946a1ae in term 1.
I20260924 03:08:33.910916 1311 leader_election.cc:302] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809b1c85ffc541cc911218e35527978f, 86646fecc8914e27b6860fe81946a1ae; no voters:
I20260924 03:08:33.911016 1442 raft_consensus.cc:2781] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:33.911155 1442 raft_consensus.cc:686] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [term 1 LEADER]: Becoming Leader. State: Replica: 86646fecc8914e27b6860fe81946a1ae, State: Running, Role: LEADER
I20260924 03:08:33.911252 1442 consensus_queue.cc:237] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } }
I20260924 03:08:33.911994 962 catalog_manager.cc:5495] T 6f263a63f0344ca9a7939e1aca32cbe4 P 86646fecc8914e27b6860fe81946a1ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 86646fecc8914e27b6860fe81946a1ae (127.0.98.3). New cstate: current_term: 1 leader_uuid: "86646fecc8914e27b6860fe81946a1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d10cbc7f774a8bb224a5e89d492941" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86646fecc8914e27b6860fe81946a1ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809b1c85ffc541cc911218e35527978f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:33.922351 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.916052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.1:36441)
0924 03:08:33.916113 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.916115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.916198 (+ 83us) server_negotiation.cc:407] Connection header received
0924 03:08:33.916246 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.916246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.916255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.916308 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.916782 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.917198 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.917803 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.917960 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.918002 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.920178 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.920183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.920184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.920189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.921462 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.921734 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.921736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.921736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.921781 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.921908 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.921908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.921908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.922031 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.922119 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.922217 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.922280 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 03:08:33.939055 1155 tablet.cc:2369] T 6f263a63f0344ca9a7939e1aca32cbe4 P 49d10cbc7f774a8bb224a5e89d492941: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:33.942170 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.935770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.98.1:36441)
0924 03:08:33.935855 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.935857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.935928 (+ 71us) server_negotiation.cc:407] Connection header received
0924 03:08:33.936018 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.936019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.936028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.936097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.936565 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.936991 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.937697 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.937880 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.937952 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.939853 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.939854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.939854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.939860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.941047 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.941398 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.941399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.941399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.941445 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.941641 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.941642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.941642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.941732 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.941824 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.941989 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.942072 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:33 dist-test-slave-jtnz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:33.960916 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:33.955251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.0.98.1:36441)
0924 03:08:33.955312 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:33.955313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:33.955361 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:08:33.955395 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:33.955396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:33.955403 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:33.955444 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:33.955910 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.956308 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.956902 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:33.957069 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:33.957127 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:33.959034 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:33.959035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:33.959035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:33.959040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:33.960030 (+ 990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.960361 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.960362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.960362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.960414 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:33.960563 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:33.960563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:33.960563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:33.960664 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:33.960726 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:33.960799 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:08:33.960844 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 03:08:33.961355 1068 tablet_service.cc:2908] Rejecting scan request for tablet 6f263a63f0344ca9a7939e1aca32cbe4: Uninitialized: clean time has not yet been initialized
I20260924 03:08:33.962568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1025
I20260924 03:08:33.967245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1159
I20260924 03:08:33.972407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1293
I20260924 03:08:33.976778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 931
2026-09-24T03:08:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1891 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:34 dist-test-slave-jtnz krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:34 dist-test-slave-jtnz krb5kdc[1460](info): set up 2 sockets
Sep 24 03:08:34 dist-test-slave-jtnz krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260924 03:08:34.961315 392 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:35.062951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:35.133385 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:35.133543 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:35.133562 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:35.134604 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:35.134636 1473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:08:35.134648 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:35.134660 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:35.134704 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:35.134721 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:35.136006 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45451
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:35.136228 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:35.136405 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:35.140782 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.140774 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.140774 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:35.140805 1473 server_base.cc:1034] running on GCE node
I20260924 03:08:35.141135 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:35.141420 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:35.142570 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790219315142559 us; error 32 us; skew 500 ppm
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:35.144654 1473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:35.145031 1473 webserver.cc:460] Webserver started at http://127.0.98.62:45959/ using document root <none> and password file <none>
I20260924 03:08:35.145181 1473 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:35.145216 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:35.145309 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:35.146157 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "c7357ca031e14b35857b5372085979f0"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "5257036aafd9360b60b8c9188a884f39"
server_key_iv: "6c27dd7a6a67b5c264ae19ae63ba8614"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.146485 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7357ca031e14b35857b5372085979f0"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "5257036aafd9360b60b8c9188a884f39"
server_key_iv: "6c27dd7a6a67b5c264ae19ae63ba8614"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.147603 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:35.148214 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.148401 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:35.148476 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "c7357ca031e14b35857b5372085979f0"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "5257036aafd9360b60b8c9188a884f39"
server_key_iv: "6c27dd7a6a67b5c264ae19ae63ba8614"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.148520 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:35.156709 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:35.158219 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:35.158331 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:35.162326 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45451
I20260924 03:08:35.162369 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45451 every 8 connection(s)
I20260924 03:08:35.162642 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:35.163200 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:35.165133 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Bootstrap starting.
I20260924 03:08:35.165632 1551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:35.165887 1551 log.cc:824] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:35.166416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1473
I20260924 03:08:35.166507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:35.166663 392 external_mini_cluster.cc:1428] Setting key 787d294085f31c214a92e332a0a26513
I20260924 03:08:35.166685 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: No bootstrap required, opened a new log
I20260924 03:08:35.168869 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260924 03:08:35.168980 1551 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:35.168999 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7357ca031e14b35857b5372085979f0, State: Initialized, Role: FOLLOWER
I20260924 03:08:35.169132 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260924 03:08:35.169203 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:35.169243 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:35.169288 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:35.169759 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260924 03:08:35.169836 1551 leader_election.cc:302] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7357ca031e14b35857b5372085979f0; no voters:
I20260924 03:08:35.170020 1551 leader_election.cc:288] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:35.170044 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:35.170166 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [term 1 LEADER]: Becoming Leader. State: Replica: c7357ca031e14b35857b5372085979f0, State: Running, Role: LEADER
I20260924 03:08:35.170272 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:35.170320 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260924 03:08:35.170718 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7357ca031e14b35857b5372085979f0. Latest consensus state: current_term: 1 leader_uuid: "c7357ca031e14b35857b5372085979f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } } }
I20260924 03:08:35.170867 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:35.171265 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:35.171653 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7357ca031e14b35857b5372085979f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7357ca031e14b35857b5372085979f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } } }
I20260924 03:08:35.171773 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:35.171837 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:35.173702 1565 catalog_manager.cc:1347] Generated new cluster ID: 0c42e5ec5cb742e3a50084decbfbdd0c
I20260924 03:08:35.173907 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:35.176908 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.167180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.98.62:45451)
0924 03:08:35.167372 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.167380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.167411 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:08:35.167885 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.167891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.167936 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.168069 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.168640 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.169645 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.170632 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.171228 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.171325 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.173803 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.173805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.173805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.173813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.175310 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.175701 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.175704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.175704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.175783 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.175991 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.175992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.175992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.176202 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.176379 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.176464 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.176590 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260924 03:08:35.183059 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:35.183692 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:35.188161 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Generated new TSK 0
I20260924 03:08:35.188319 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:35.214411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45451
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:35.303965 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:35.304134 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:35.304154 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:35.305253 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:35.305289 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:35.305307 1577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:35.305322 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:35.306674 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:35.306870 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:35.307086 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:35.312601 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.312613 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:35.312726 1577 server_base.cc:1034] running on GCE node
W20260924 03:08:35.312850 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:35.313005 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:35.313176 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:35.314302 1577 hybrid_clock.cc:648] HybridClock initialized: now 1790219315314291 us; error 26 us; skew 500 ppm
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:35.316562 1577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:35.316959 1577 webserver.cc:460] Webserver started at http://127.0.98.1:45285/ using document root <none> and password file <none>
I20260924 03:08:35.317160 1577 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:35.317219 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:35.317332 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:35.318387 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "98785734dfc742459d93ee283154aee1"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "e8f2ed5691a82297cc977b35c06c33ee"
server_key_iv: "1a580891ac12bb2d5c0de2093c01147f"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.318712 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98785734dfc742459d93ee283154aee1"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "e8f2ed5691a82297cc977b35c06c33ee"
server_key_iv: "1a580891ac12bb2d5c0de2093c01147f"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.319823 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:35.320433 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.320595 1577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:35.320688 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "98785734dfc742459d93ee283154aee1"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "e8f2ed5691a82297cc977b35c06c33ee"
server_key_iv: "1a580891ac12bb2d5c0de2093c01147f"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.320756 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:35.339005 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:35.340958 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:35.341099 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:35.341548 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:35.341604 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.341656 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:35.341687 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.347651 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46813
I20260924 03:08:35.347780 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46813 every 8 connection(s)
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:35.348047 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:35.348628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1577
I20260924 03:08:35.348718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:35.348881 392 external_mini_cluster.cc:1428] Setting key c2d8c77cbb8208bde6bd511fea4619c4
I20260924 03:08:35.351330 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.341814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41171 (local address 127.0.98.62:45451)
0924 03:08:35.341868 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.341870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.342110 (+ 240us) server_negotiation.cc:407] Connection header received
0924 03:08:35.342584 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.342586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.342606 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.342680 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.343388 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.344159 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.345577 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.345885 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.345953 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.348473 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.348474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.348475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.348481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.349817 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.350291 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.350292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.350292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.350341 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.350674 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.350675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.350676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.350819 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.350913 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.351166 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.351247 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:08:35.351471 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.341817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.1:41171)
0924 03:08:35.341957 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.342002 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.342494 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.342739 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.342742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.343002 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.343283 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.343289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.344272 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.344273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.345483 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.345486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.345963 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.345963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.346078 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.346549 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.346558 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.348374 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.349891 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.349897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.349898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.350201 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.350413 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.350414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.350414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.350580 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.350888 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.350895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.351022 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.351130 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.351269 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 03:08:35.352054 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45451
I20260924 03:08:35.352139 1708 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:35.352329 1708 heartbeater.cc:511] Master 127.0.98.62:45451 requested a full tablet report, sending...
I20260924 03:08:35.352798 1504 ts_manager.cc:194] Registered new tserver with Master: 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813)
I20260924 03:08:35.353669 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41171
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:35.358647 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.353038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57863 (local address 127.0.98.62:45451)
0924 03:08:35.353079 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.353080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.353178 (+ 98us) server_negotiation.cc:407] Connection header received
0924 03:08:35.353239 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.353240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.353249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.353298 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.353596 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.354250 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.355235 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.355387 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.355435 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.356426 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.356427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.356427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.356432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.357745 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.358028 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.358029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.358029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.358079 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.358277 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.358278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.358278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.358378 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.358426 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.358545 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.358595 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:35.358656 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.352997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.1:57863)
0924 03:08:35.353113 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.353121 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.353189 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.353302 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.353303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.353464 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.353510 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.353512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.354332 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.354333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.355152 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.355155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.355457 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.355458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.355529 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.355923 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.355928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.356335 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.357817 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.357818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.357818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.357968 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.358139 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.358140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.358140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.358222 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.358440 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.358440 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.358480 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.358530 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.358595 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:35.372965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45451
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:35.445494 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:35.445688 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:35.445724 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:35.446864 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:35.446924 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:35.446956 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:35.446998 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:35.448432 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:35.448678 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:35.448880 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:35.453563 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.453588 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.453588 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:35.453685 1717 server_base.cc:1034] running on GCE node
I20260924 03:08:35.453922 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:35.454146 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:35.455350 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790219315455327 us; error 38 us; skew 500 ppm
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:35.457613 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:35.457980 1717 webserver.cc:460] Webserver started at http://127.0.98.2:43913/ using document root <none> and password file <none>
I20260924 03:08:35.458158 1717 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:35.458253 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:35.458410 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:35.459373 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "f33b3a5b02a34b2496f2d8296e141d01"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "1bab689d9d5edd3c22c1d8778d589547"
server_key_iv: "b1f16f2e55c299486c00451d9dfa20be"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.459699 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f33b3a5b02a34b2496f2d8296e141d01"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "1bab689d9d5edd3c22c1d8778d589547"
server_key_iv: "b1f16f2e55c299486c00451d9dfa20be"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.460865 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 03:08:35.461517 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.461681 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:35.461760 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "f33b3a5b02a34b2496f2d8296e141d01"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "1bab689d9d5edd3c22c1d8778d589547"
server_key_iv: "b1f16f2e55c299486c00451d9dfa20be"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.461828 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:35.481216 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:35.482910 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:35.483062 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:35.483543 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:35.483585 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.483606 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:35.483618 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:35.490631 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35449
I20260924 03:08:35.490885 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35449 every 8 connection(s)
I20260924 03:08:35.490981 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:35.493076 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.484032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43343 (local address 127.0.98.62:45451)
0924 03:08:35.484109 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.484111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.484236 (+ 125us) server_negotiation.cc:407] Connection header received
0924 03:08:35.484635 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.484636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.484651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.484728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.485517 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.486347 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.487523 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.487737 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.487812 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.490375 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.490376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.490376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.490384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.491834 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.492216 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.492218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.492218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.492270 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.492513 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.492514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.492514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.492623 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.492688 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.492924 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.492996 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:08:35.493255 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.483960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.2:43343)
0924 03:08:35.484125 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.484160 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.484532 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.484815 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.484818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.485147 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.485419 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.485425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.486457 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.486458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.487416 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.487418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.487842 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.487844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.487950 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.488448 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.488458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.490278 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.491939 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.491945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.491946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.492144 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.492339 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.492340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.492340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.492439 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.492692 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.492698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.492813 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.492906 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.493038 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260924 03:08:35.493667 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45451
I20260924 03:08:35.493741 1848 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:35.493903 1848 heartbeater.cc:511] Master 127.0.98.62:45451 requested a full tablet report, sending...
I20260924 03:08:35.494236 1505 ts_manager.cc:194] Registered new tserver with Master: f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449)
I20260924 03:08:35.494544 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43343
I20260924 03:08:35.496994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1717
I20260924 03:08:35.497093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:35.497290 392 external_mini_cluster.cc:1428] Setting key 318142b7b774f71608ebf25da772bf6d
I20260924 03:08:35.499604 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.494757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42075 (local address 127.0.98.62:45451)
0924 03:08:35.494790 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.494792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.494799 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:35.494817 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.494818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.494843 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.494901 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.495214 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.495604 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.496173 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.496320 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.496362 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.497130 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.497131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.497132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.497137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.498536 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.498858 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.498859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.498859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.498905 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.499086 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.499086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.499087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.499215 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.499299 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.499449 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.499515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:35.499610 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.494627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.2:42075)
0924 03:08:35.494729 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.494738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.494780 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.494895 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.494896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.495070 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.495145 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.495147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.495681 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.495682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.496113 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.496114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.496374 (+ 260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.496374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.496415 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.496787 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.496792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.497027 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.498632 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.498632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.498633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.498782 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.498971 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.498971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.498972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.499021 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.499304 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.499305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.499344 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.499424 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.499514 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:35.520157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45451
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:35.592711 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:35.592893 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:35.592916 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:35.594039 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:35.594080 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:35.594105 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:35.594122 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:35.595557 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:35.595723 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:35.595877 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:35.600401 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.600410 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:35.600472 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:35.600608 1857 server_base.cc:1034] running on GCE node
I20260924 03:08:35.600816 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:35.601019 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:35.602231 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790219315602214 us; error 29 us; skew 500 ppm
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:35.604521 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:35.604995 1857 webserver.cc:460] Webserver started at http://127.0.98.3:42879/ using document root <none> and password file <none>
I20260924 03:08:35.605162 1857 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:35.605201 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:35.605280 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:35.606163 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f6df65bdccd45a1a0700372bcc4b998"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "c2ec61ec79b5d1b8b2bfaa44cbdf7414"
server_key_iv: "0d116d65482708fce4017e3f2664c51a"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.606511 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f6df65bdccd45a1a0700372bcc4b998"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "c2ec61ec79b5d1b8b2bfaa44cbdf7414"
server_key_iv: "0d116d65482708fce4017e3f2664c51a"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.607630 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:08:35.608237 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.608381 1857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:35.608469 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "0f6df65bdccd45a1a0700372bcc4b998"
format_stamp: "Formatted at 2026-09-24 03:08:35 on dist-test-slave-jtnz"
server_key: "c2ec61ec79b5d1b8b2bfaa44cbdf7414"
server_key_iv: "0d116d65482708fce4017e3f2664c51a"
server_key_version: "encryptionkey@0"
I20260924 03:08:35.608516 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:35.630517 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:35.632506 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:35.632599 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:35.632973 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:35.633009 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:35.633028 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:35.633039 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:35.639439 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38779
I20260924 03:08:35.639482 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38779 every 8 connection(s)
I20260924 03:08:35.639844 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:35.642244 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.633477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40189 (local address 127.0.98.62:45451)
0924 03:08:35.633541 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.633543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.633697 (+ 154us) server_negotiation.cc:407] Connection header received
0924 03:08:35.634117 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.634118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.634129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.634191 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.634997 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.635812 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.637228 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.637580 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.637666 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.639777 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.639778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.639779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.639785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.641162 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.641521 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.641522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.641522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.641564 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.641723 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.641723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.641723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.641802 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.641844 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.642066 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.642122 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:08:35.642401 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.633444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.3:40189)
0924 03:08:35.633593 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.633632 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.634021 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.634255 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.634258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.634609 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.634864 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.634888 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.635906 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.635907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.637125 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.637127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.637664 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.637665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.637751 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.638231 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.638239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.639688 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.641262 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.641265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.641267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.641450 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.641607 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.641607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.641607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.641681 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.641851 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.641854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.641944 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.642046 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.642152 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260924 03:08:35.642870 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45451
I20260924 03:08:35.642946 1988 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:35.643111 1988 heartbeater.cc:511] Master 127.0.98.62:45451 requested a full tablet report, sending...
I20260924 03:08:35.643450 1505 ts_manager.cc:194] Registered new tserver with Master: 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3:38779)
I20260924 03:08:35.643899 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40189
I20260924 03:08:35.644129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 1857
I20260924 03:08:35.644227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:35.644384 392 external_mini_cluster.cc:1428] Setting key e8c64bc6539ffb929895806ee1f55e3e
I20260924 03:08:35.645175 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:35.649277 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.643833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45869 (local address 127.0.98.62:45451)
0924 03:08:35.643878 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.643880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.643997 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:08:35.644059 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.644060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.644068 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.644125 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.644446 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.645070 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.645774 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.646044 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.646112 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.646917 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.646918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.646918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.646924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.648283 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.648669 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.648670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.648670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.648721 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.648908 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.648908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.648908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.648987 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.649028 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.649183 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.649228 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:35.649420 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.643779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.98.3:45869)
0924 03:08:35.643915 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.643929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.644008 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.644129 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.644130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.644326 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:35.644375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.644376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.645178 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.645179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.645667 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.645669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.646119 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.646119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.646160 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.646542 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:35.646545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:35.646818 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:35.648369 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.648370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.648371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.648593 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.648789 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:35.648790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:35.648790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:35.648855 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:35.649043 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:35.649044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:35.649085 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.649168 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.649258 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:35 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:35.675913 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.668944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.0.98.62:45451)
0924 03:08:35.669006 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.669008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.669027 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:08:35.669050 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.669050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.669060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.669121 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:35.669864 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.670523 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.671458 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.671732 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.671810 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.673793 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:35.673795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:35.673795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:35.673802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:35.674910 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.675210 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.675211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.675212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.675264 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:35.675456 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:35.675457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:35.675457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:35.675560 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:35.675654 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.675758 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.675824 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:08:35.677368 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:35.678992 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:35.685045 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.681108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.0.98.2:35449)
0924 03:08:35.681372 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.681377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.681388 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:35.681426 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.681428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.681445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.681529 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:35.681717 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682842 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684019 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.684639 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684701 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.684776 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.684908 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.684949 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260924 03:08:35.685058 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.680893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38779 (local address 127.0.0.1:54072)
0924 03:08:35.681158 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.681167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.681241 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.681596 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.681597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.681599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:35.681652 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.681654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682626 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.682627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.683873 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.683875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.684711 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.684785 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.684798 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.684889 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.684950 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":26,"thread_start_us":47,"threads_started":1}
I20260924 03:08:35.685292 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.681266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.98.3:38779)
0924 03:08:35.681428 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.681431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.681440 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:35.681486 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.681488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.681501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.681724 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:35.681777 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682524 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.683956 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.684629 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684701 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.684772 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.685173 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.685210 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260924 03:08:35.685057 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.680886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.0.1:57986)
0924 03:08:35.681107 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.681119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.681173 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.681554 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.681557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.681560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:35.681632 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.681634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682928 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.682929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.683952 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.683954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.684710 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.684791 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.684798 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.684886 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.684950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":25,"thread_start_us":47,"threads_started":1}
I20260924 03:08:35.685834 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.680848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46813 (local address 127.0.0.1:45470)
0924 03:08:35.681037 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.681075 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.681145 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.681656 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.681657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.681659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:35.681698 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.681699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682973 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.682973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.683885 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.683886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.685641 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.685642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.685695 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.685699 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.685742 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.685772 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260924 03:08:35.685868 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2f62b8173b48c0a31911207769cbae (DEFAULT_TABLE table=test-table [id=34ed3f1b377f42d39cd71a41516109e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:35.685906 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2f62b8173b48c0a31911207769cbae (DEFAULT_TABLE table=test-table [id=34ed3f1b377f42d39cd71a41516109e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:35.686300 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.680927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.98.1:46813)
0924 03:08:35.681424 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.681428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.681437 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:35.681466 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.681468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.681480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.681773 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:35.681832 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.682877 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.684808 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.685550 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.686002 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.686073 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.686108 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.686143 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":108,"threads_started":1}
I20260924 03:08:35.686306 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2f62b8173b48c0a31911207769cbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:35.686307 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2f62b8173b48c0a31911207769cbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:35.688920 2008 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Bootstrap starting.
I20260924 03:08:35.689267 2009 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Bootstrap starting.
I20260924 03:08:35.689582 2008 tablet_bootstrap.cc:654] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:35.689663 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2f62b8173b48c0a31911207769cbae (DEFAULT_TABLE table=test-table [id=34ed3f1b377f42d39cd71a41516109e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:35.689852 2009 tablet_bootstrap.cc:654] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:35.689895 2008 log.cc:824] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:35.689976 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2f62b8173b48c0a31911207769cbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:35.690131 2009 log.cc:824] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:35.690738 2008 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: No bootstrap required, opened a new log
I20260924 03:08:35.690809 2008 ts_tablet_manager.cc:1397] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:35.692765 2008 raft_consensus.cc:348] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.692911 2008 raft_consensus.cc:374] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:35.692942 2008 raft_consensus.cc:729] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6df65bdccd45a1a0700372bcc4b998, State: Initialized, Role: FOLLOWER
I20260924 03:08:35.693064 2008 consensus_queue.cc:260] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.693313 2008 ts_tablet_manager.cc:1428] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:08:35.693344 1988 heartbeater.cc:503] Master 127.0.98.62:45451 was elected leader, sending a full tablet report...
I20260924 03:08:35.693449 2013 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Bootstrap starting.
I20260924 03:08:35.694023 2013 tablet_bootstrap.cc:654] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:35.694322 2009 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: No bootstrap required, opened a new log
I20260924 03:08:35.694389 2009 ts_tablet_manager.cc:1397] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 03:08:35.694377 2013 log.cc:824] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:35.694972 2013 tablet_bootstrap.cc:492] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: No bootstrap required, opened a new log
I20260924 03:08:35.695016 2013 ts_tablet_manager.cc:1397] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:08:35.695822 2009 raft_consensus.cc:348] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.695955 2009 raft_consensus.cc:374] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:35.696007 2009 raft_consensus.cc:729] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33b3a5b02a34b2496f2d8296e141d01, State: Initialized, Role: FOLLOWER
I20260924 03:08:35.696132 2009 consensus_queue.cc:260] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.696381 2009 ts_tablet_manager.cc:1428] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:08:35.696302 2013 raft_consensus.cc:348] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.696420 1848 heartbeater.cc:503] Master 127.0.98.62:45451 was elected leader, sending a full tablet report...
I20260924 03:08:35.696429 2013 raft_consensus.cc:374] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:35.696461 2013 raft_consensus.cc:729] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98785734dfc742459d93ee283154aee1, State: Initialized, Role: FOLLOWER
I20260924 03:08:35.696549 2013 consensus_queue.cc:260] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.696764 2013 ts_tablet_manager.cc:1428] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:08:35.696830 1708 heartbeater.cc:503] Master 127.0.98.62:45451 was elected leader, sending a full tablet report...
W20260924 03:08:35.741524 1849 tablet.cc:2369] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:35.848703 1709 tablet.cc:2369] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:35.890434 1989 tablet.cc:2369] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:35.987028 2012 raft_consensus.cc:482] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:35.987149 2012 raft_consensus.cc:504] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.987651 2012 leader_election.cc:288] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449), 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813)
I20260924 03:08:35.991434 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.987913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59355 (local address 127.0.98.1:46813)
0924 03:08:35.987984 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.987986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.988018 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:08:35.988050 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.988051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.988068 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.988140 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:35.988338 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.989089 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.990557 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.991093 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.991158 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.991167 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.991336 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.991371 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:08:35.991433 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.987921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44037 (local address 127.0.98.2:35449)
0924 03:08:35.987984 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:35.987986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:35.988104 (+ 118us) server_negotiation.cc:407] Connection header received
0924 03:08:35.988151 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:35.988152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:35.988164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:35.988220 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:35.988392 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.989089 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.990575 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.991085 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.991158 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.991167 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:35.991336 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 03:08:35.991368 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:08:35.991447 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.987856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46813 (local address 127.0.98.3:59355)
0924 03:08:35.987912 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.987924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.987966 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.988164 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.988165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.988169 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:35.988263 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.988264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.989170 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.989171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.990473 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.990475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.991168 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.991169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.991256 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.991262 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.991317 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.991359 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":109}
I20260924 03:08:35.991456 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:35.987857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.98.3:44037)
0924 03:08:35.988031 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:35.988047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:35.988087 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:35.988232 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:35.988233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:35.988234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:35.988310 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.988312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.989170 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.989171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.990473 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:35.990475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:35.991168 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:35.991169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:35.991256 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:35.991262 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:08:35.991317 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:08:35.991359 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":99,"thread_start_us":96,"threads_started":1}
I20260924 03:08:35.991807 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2f62b8173b48c0a31911207769cbae" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1" is_pre_election: true
I20260924 03:08:35.991956 1662 raft_consensus.cc:2454] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 0.
I20260924 03:08:35.992004 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2f62b8173b48c0a31911207769cbae" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33b3a5b02a34b2496f2d8296e141d01" is_pre_election: true
I20260924 03:08:35.992169 1802 raft_consensus.cc:2454] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 0.
I20260924 03:08:35.992203 1874 leader_election.cc:302] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6df65bdccd45a1a0700372bcc4b998, 98785734dfc742459d93ee283154aee1; no voters:
I20260924 03:08:35.992329 2012 raft_consensus.cc:2781] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:35.992417 2012 raft_consensus.cc:482] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:35.992445 2012 raft_consensus.cc:3037] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:35.993429 2012 raft_consensus.cc:504] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.993546 2012 leader_election.cc:288] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 election: Requested vote from peers f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449), 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813)
I20260924 03:08:35.993681 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2f62b8173b48c0a31911207769cbae" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33b3a5b02a34b2496f2d8296e141d01"
I20260924 03:08:35.993680 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2f62b8173b48c0a31911207769cbae" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1"
I20260924 03:08:35.993749 1662 raft_consensus.cc:3037] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:35.993749 1802 raft_consensus.cc:3037] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:35.995074 1802 raft_consensus.cc:2454] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 1.
I20260924 03:08:35.995237 1662 raft_consensus.cc:2454] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 1.
I20260924 03:08:35.995397 1874 leader_election.cc:302] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6df65bdccd45a1a0700372bcc4b998, 98785734dfc742459d93ee283154aee1; no voters:
I20260924 03:08:35.995479 2012 raft_consensus.cc:2781] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:35.995626 2012 raft_consensus.cc:686] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6df65bdccd45a1a0700372bcc4b998, State: Running, Role: LEADER
I20260924 03:08:35.995729 2012 consensus_queue.cc:237] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:35.996507 1505 catalog_manager.cc:5495] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f6df65bdccd45a1a0700372bcc4b998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } health_report { overall_health: HEALTHY } } }
I20260924 03:08:36.039677 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.037513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.0.98.3:38779)
0924 03:08:36.037550 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.037552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.037559 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:36.037649 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.037649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.037663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.037712 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:36.037823 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.038263 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.038953 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.039151 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.039198 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.039365 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.039445 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.039500 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.039543 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.039560 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:36.041273 1802 raft_consensus.cc:1261] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Refusing update from remote peer 0f6df65bdccd45a1a0700372bcc4b998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.041273 1662 raft_consensus.cc:1261] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Refusing update from remote peer 0f6df65bdccd45a1a0700372bcc4b998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.041563 2018 consensus_queue.cc:1035] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.041692 2012 consensus_queue.cc:1035] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.045214 2023 mvcc.cc:205] Tried to move back new op lower bound from 7332738318502277120 to 7332738318319022080. Current Snapshot: MvccSnapshot[applied={T|T < 7332738318502277120 or (T in {7332738318502277120})}]
I20260924 03:08:36.045349 2022 mvcc.cc:205] Tried to move back new op lower bound from 7332738318502277120 to 7332738318319022080. Current Snapshot: MvccSnapshot[applied={T|T < 7332738318502277120 or (T in {7332738318502277120})}]
I20260924 03:08:36.046926 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:36.046998 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:36.047757 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Sending DeleteTablet for 3 replicas of tablet 5c2f62b8173b48c0a31911207769cbae
I20260924 03:08:36.048061 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2f62b8173b48c0a31911207769cbae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54072
I20260924 03:08:36.048132 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2f62b8173b48c0a31911207769cbae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57986
I20260924 03:08:36.048276 2031 tablet_replica.cc:330] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: stopping tablet replica
I20260924 03:08:36.048319 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2f62b8173b48c0a31911207769cbae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45470
I20260924 03:08:36.048374 2031 raft_consensus.cc:2229] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:36.048344 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:08:36.048470 2031 raft_consensus.cc:2258] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:08:36.048489 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:36.048897 2032 tablet_replica.cc:330] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: stopping tablet replica
I20260924 03:08:36.048920 2031 ts_tablet_manager.cc:1905] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.049016 2032 raft_consensus.cc:2229] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:36.049126 2032 raft_consensus.cc:2258] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:36.049261 2033 tablet_replica.cc:330] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: stopping tablet replica
I20260924 03:08:36.049353 2033 raft_consensus.cc:2229] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:36.049446 2033 raft_consensus.cc:2258] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:36.049830 2033 ts_tablet_manager.cc:1905] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.050550 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 60e66beec5904777be8c5b64d8248a47 (DEFAULT_TABLE table=test-table [id=be762458f56d49ab93affd6865085a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:36.050735 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e66beec5904777be8c5b64d8248a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.050735 2031 ts_tablet_manager.cc:1918] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:36.050827 2031 log.cc:1196] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/5c2f62b8173b48c0a31911207769cbae
I20260924 03:08:36.051149 2031 ts_tablet_manager.cc:1939] T 5c2f62b8173b48c0a31911207769cbae P 0f6df65bdccd45a1a0700372bcc4b998: Deleting consensus metadata
I20260924 03:08:36.051424 2033 ts_tablet_manager.cc:1918] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:36.051476 2033 log.cc:1196] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/5c2f62b8173b48c0a31911207769cbae
I20260924 03:08:36.051690 2033 ts_tablet_manager.cc:1939] T 5c2f62b8173b48c0a31911207769cbae P 98785734dfc742459d93ee283154aee1: Deleting consensus metadata
I20260924 03:08:36.051870 2008 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Bootstrap starting.
I20260924 03:08:36.052143 1490 catalog_manager.cc:4841] TS 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813): tablet 5c2f62b8173b48c0a31911207769cbae (table test-table [id=34ed3f1b377f42d39cd71a41516109e1]) successfully deleted
I20260924 03:08:36.052352 2008 tablet_bootstrap.cc:654] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.052395 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 60e66beec5904777be8c5b64d8248a47 (DEFAULT_TABLE table=test-table [id=be762458f56d49ab93affd6865085a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:36.052484 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e66beec5904777be8c5b64d8248a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.052939 2008 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: No bootstrap required, opened a new log
I20260924 03:08:36.053002 2008 ts_tablet_manager.cc:1397] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:36.053143 2008 raft_consensus.cc:348] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.053227 2008 raft_consensus.cc:374] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.053256 2008 raft_consensus.cc:729] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6df65bdccd45a1a0700372bcc4b998, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.053301 2008 consensus_queue.cc:260] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.053373 2008 ts_tablet_manager.cc:1428] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:36.053656 2013 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Bootstrap starting.
I20260924 03:08:36.053988 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 60e66beec5904777be8c5b64d8248a47 (DEFAULT_TABLE table=test-table [id=be762458f56d49ab93affd6865085a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:36.054025 2013 tablet_bootstrap.cc:654] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.054100 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e66beec5904777be8c5b64d8248a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.054291 1489 catalog_manager.cc:4841] TS 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3:38779): tablet 5c2f62b8173b48c0a31911207769cbae (table test-table [id=34ed3f1b377f42d39cd71a41516109e1]) successfully deleted
I20260924 03:08:36.054703 2032 ts_tablet_manager.cc:1905] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.055380 2009 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Bootstrap starting.
I20260924 03:08:36.055748 2009 tablet_bootstrap.cc:654] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.056145 2013 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: No bootstrap required, opened a new log
I20260924 03:08:36.056143 2032 ts_tablet_manager.cc:1918] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:36.056193 2013 ts_tablet_manager.cc:1397] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 03:08:36.056202 2032 log.cc:1196] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/5c2f62b8173b48c0a31911207769cbae
I20260924 03:08:36.056352 2013 raft_consensus.cc:348] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.056432 2013 raft_consensus.cc:374] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.056450 2013 raft_consensus.cc:729] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98785734dfc742459d93ee283154aee1, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.056470 2032 ts_tablet_manager.cc:1939] T 5c2f62b8173b48c0a31911207769cbae P f33b3a5b02a34b2496f2d8296e141d01: Deleting consensus metadata
I20260924 03:08:36.056514 2013 consensus_queue.cc:260] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.056612 2013 ts_tablet_manager.cc:1428] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:36.056910 2009 tablet_bootstrap.cc:492] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: No bootstrap required, opened a new log
I20260924 03:08:36.056898 1492 catalog_manager.cc:4841] TS f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449): tablet 5c2f62b8173b48c0a31911207769cbae (table test-table [id=34ed3f1b377f42d39cd71a41516109e1]) successfully deleted
I20260924 03:08:36.056953 2009 ts_tablet_manager.cc:1397] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:08:36.057293 2009 raft_consensus.cc:348] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.057366 2009 raft_consensus.cc:374] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.057381 2009 raft_consensus.cc:729] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33b3a5b02a34b2496f2d8296e141d01, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.057435 2009 consensus_queue.cc:260] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.057530 2009 ts_tablet_manager.cc:1428] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:08:36.474395 2018 raft_consensus.cc:482] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:36.474485 2018 raft_consensus.cc:504] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.474673 2018 leader_election.cc:288] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813), f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449)
I20260924 03:08:36.474938 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e66beec5904777be8c5b64d8248a47" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33b3a5b02a34b2496f2d8296e141d01" is_pre_election: true
I20260924 03:08:36.474937 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e66beec5904777be8c5b64d8248a47" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1" is_pre_election: true
I20260924 03:08:36.475036 1662 raft_consensus.cc:2454] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 0.
I20260924 03:08:36.475036 1802 raft_consensus.cc:2454] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 0.
I20260924 03:08:36.475184 1874 leader_election.cc:302] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6df65bdccd45a1a0700372bcc4b998, 98785734dfc742459d93ee283154aee1; no voters:
I20260924 03:08:36.475329 2018 raft_consensus.cc:2781] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:36.475389 2018 raft_consensus.cc:482] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:36.475421 2018 raft_consensus.cc:3037] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.476147 2018 raft_consensus.cc:504] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.476310 2018 leader_election.cc:288] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 election: Requested vote from peers 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813), f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449)
I20260924 03:08:36.476475 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e66beec5904777be8c5b64d8248a47" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1"
I20260924 03:08:36.476521 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e66beec5904777be8c5b64d8248a47" candidate_uuid: "0f6df65bdccd45a1a0700372bcc4b998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33b3a5b02a34b2496f2d8296e141d01"
I20260924 03:08:36.476563 1662 raft_consensus.cc:3037] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.476600 1802 raft_consensus.cc:3037] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.477311 1802 raft_consensus.cc:2454] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 1.
I20260924 03:08:36.477311 1662 raft_consensus.cc:2454] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6df65bdccd45a1a0700372bcc4b998 in term 1.
I20260924 03:08:36.477485 1874 leader_election.cc:302] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6df65bdccd45a1a0700372bcc4b998, 98785734dfc742459d93ee283154aee1; no voters:
I20260924 03:08:36.477584 2018 raft_consensus.cc:2781] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:36.477643 2018 raft_consensus.cc:686] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6df65bdccd45a1a0700372bcc4b998, State: Running, Role: LEADER
I20260924 03:08:36.477694 2018 consensus_queue.cc:237] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.478322 1504 catalog_manager.cc:5495] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f6df65bdccd45a1a0700372bcc4b998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } health_report { overall_health: HEALTHY } } }
Sep 24 03:08:36 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:36.516916 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.510750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45451 (local address 127.0.0.1:53176)
0924 03:08:36.510959 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.510974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.511028 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.511202 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.511202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.511459 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:36.511499 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.511500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.512129 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.512130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.512628 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.512630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.512926 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.512927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.512962 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.513464 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:36.513468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:36.514741 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:36.516048 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:36.516054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:36.516055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:36.516248 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:36.516403 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:36.516404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:36.516404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:36.516453 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:36.516653 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:36.516658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:36.516718 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.516763 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.516837 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
I20260924 03:08:36.517014 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.510840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.0.98.62:45451)
0924 03:08:36.510948 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.510950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.511050 (+ 100us) server_negotiation.cc:407] Connection header received
0924 03:08:36.511096 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.511097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.511108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.511179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:36.511560 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.512041 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.512695 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.512868 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.512915 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.514823 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:36.514824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:36.514825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:36.514830 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:36.515982 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.516305 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.516306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.516306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.516353 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.516517 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.516517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.516517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.516603 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:36.516650 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.516793 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.516879 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260924 03:08:36.518657 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:08:36.518836 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:36.520092 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 4c19d7dac10346e1bfd9883b9087b386 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d701a53ec37c45df8643bbdeee76148e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:36.520107 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 4c19d7dac10346e1bfd9883b9087b386 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d701a53ec37c45df8643bbdeee76148e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:36.520125 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 4c19d7dac10346e1bfd9883b9087b386 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d701a53ec37c45df8643bbdeee76148e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:36.520246 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c19d7dac10346e1bfd9883b9087b386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.520246 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c19d7dac10346e1bfd9883b9087b386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.520257 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c19d7dac10346e1bfd9883b9087b386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:36.521217 2008 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: Bootstrap starting.
I20260924 03:08:36.521598 2008 tablet_bootstrap.cc:654] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.521600 2009 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: Bootstrap starting.
I20260924 03:08:36.521596 2013 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: Bootstrap starting.
I20260924 03:08:36.521875 2009 tablet_bootstrap.cc:654] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.521951 2013 tablet_bootstrap.cc:654] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:36.522128 2008 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: No bootstrap required, opened a new log
I20260924 03:08:36.522159 2008 ts_tablet_manager.cc:1397] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:36.522313 2008 raft_consensus.cc:348] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.522357 2008 raft_consensus.cc:374] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.522370 2008 raft_consensus.cc:729] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6df65bdccd45a1a0700372bcc4b998, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.522423 2008 consensus_queue.cc:260] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.522509 2008 ts_tablet_manager.cc:1428] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:36.522522 2018 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.522580 2018 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.522995 2009 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: No bootstrap required, opened a new log
I20260924 03:08:36.523075 2009 ts_tablet_manager.cc:1397] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:08:36.523250 2013 tablet_bootstrap.cc:492] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: No bootstrap required, opened a new log
I20260924 03:08:36.523253 2009 raft_consensus.cc:348] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.523317 2013 ts_tablet_manager.cc:1397] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:08:36.523326 2009 raft_consensus.cc:374] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.523348 2009 raft_consensus.cc:729] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33b3a5b02a34b2496f2d8296e141d01, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.523396 2009 consensus_queue.cc:260] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.523449 2013 raft_consensus.cc:348] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.523485 2009 ts_tablet_manager.cc:1428] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:36.523511 2013 raft_consensus.cc:374] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:36.523535 2013 raft_consensus.cc:729] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98785734dfc742459d93ee283154aee1, State: Initialized, Role: FOLLOWER
I20260924 03:08:36.523576 2013 consensus_queue.cc:260] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.523571 2015 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.523638 2015 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.523659 2013 ts_tablet_manager.cc:1428] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:36.523725 2016 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.523806 2016 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.769405 2015 raft_consensus.cc:482] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:36.769496 2015 raft_consensus.cc:504] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.769917 2015 leader_election.cc:288] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813), 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3:38779)
I20260924 03:08:36.773835 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.770202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40631 (local address 127.0.98.1:46813)
0924 03:08:36.770305 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.770307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.770318 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:36.770353 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.770353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.770363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.770463 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:36.770629 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.771348 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.772923 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.773476 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.773541 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.773551 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.773710 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.773747 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260924 03:08:36.773828 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.770228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34121 (local address 127.0.98.3:38779)
0924 03:08:36.770313 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.770315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.770326 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:36.770353 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.770353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.770363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.770435 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:36.770597 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.771364 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.772923 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.773465 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.773541 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.773551 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.773710 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.773751 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":40,"threads_started":1}
I20260924 03:08:36.773874 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.770101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38779 (local address 127.0.98.2:34121)
0924 03:08:36.770242 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.770253 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.770306 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.770444 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.770444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.770446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:36.770538 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.770540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.771440 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.771440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.772819 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.772822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.773556 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.773556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.773626 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.773632 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.773688 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.773738 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":124,"thread_start_us":68,"threads_started":1}
I20260924 03:08:36.773874 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.770106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46813 (local address 127.0.98.2:40631)
0924 03:08:36.770253 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.770261 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.770306 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.770444 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.770444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.770448 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:36.770557 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.770559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.771439 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.771440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.772818 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.772822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.773554 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.773555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.773626 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.773632 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.773688 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.773738 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":99,"thread_start_us":64,"threads_started":1}
I20260924 03:08:36.774128 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c19d7dac10346e1bfd9883b9087b386" candidate_uuid: "f33b3a5b02a34b2496f2d8296e141d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6df65bdccd45a1a0700372bcc4b998" is_pre_election: true
I20260924 03:08:36.774298 1942 raft_consensus.cc:2454] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33b3a5b02a34b2496f2d8296e141d01 in term 0.
I20260924 03:08:36.774452 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c19d7dac10346e1bfd9883b9087b386" candidate_uuid: "f33b3a5b02a34b2496f2d8296e141d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1" is_pre_election: true
I20260924 03:08:36.774509 1733 leader_election.cc:302] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6df65bdccd45a1a0700372bcc4b998, f33b3a5b02a34b2496f2d8296e141d01; no voters:
I20260924 03:08:36.774569 1662 raft_consensus.cc:2454] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33b3a5b02a34b2496f2d8296e141d01 in term 0.
I20260924 03:08:36.774624 2015 raft_consensus.cc:2781] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:36.774664 2015 raft_consensus.cc:482] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:36.774691 2015 raft_consensus.cc:3037] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.775302 2015 raft_consensus.cc:504] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.775421 2015 leader_election.cc:288] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [CANDIDATE]: Term 1 election: Requested vote from peers 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813), 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3:38779)
I20260924 03:08:36.775564 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c19d7dac10346e1bfd9883b9087b386" candidate_uuid: "f33b3a5b02a34b2496f2d8296e141d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785734dfc742459d93ee283154aee1"
I20260924 03:08:36.775667 1662 raft_consensus.cc:3037] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.776196 1662 raft_consensus.cc:2454] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33b3a5b02a34b2496f2d8296e141d01 in term 1.
I20260924 03:08:36.776302 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c19d7dac10346e1bfd9883b9087b386" candidate_uuid: "f33b3a5b02a34b2496f2d8296e141d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6df65bdccd45a1a0700372bcc4b998"
I20260924 03:08:36.776381 1942 raft_consensus.cc:3037] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:36.776369 1734 leader_election.cc:302] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98785734dfc742459d93ee283154aee1, f33b3a5b02a34b2496f2d8296e141d01; no voters:
I20260924 03:08:36.776499 2015 raft_consensus.cc:2781] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:36.776584 2015 raft_consensus.cc:686] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 LEADER]: Becoming Leader. State: Replica: f33b3a5b02a34b2496f2d8296e141d01, State: Running, Role: LEADER
I20260924 03:08:36.776667 2015 consensus_queue.cc:237] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } }
I20260924 03:08:36.776943 2052 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: TxnStatusTablet state changed. Reason: New leader f33b3a5b02a34b2496f2d8296e141d01. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.777033 2052 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:36.777194 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:08:36.777225 1942 raft_consensus.cc:2454] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33b3a5b02a34b2496f2d8296e141d01 in term 1.
I20260924 03:08:36.777303 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:08:36.777666 1505 catalog_manager.cc:5495] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 reported cstate change: term changed from 0 to 1, leader changed from <none> to f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } health_report { overall_health: UNKNOWN } } }
I20260924 03:08:36.805040 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.802497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.0.1:60040)
0924 03:08:36.802585 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.802609 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.802664 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.802857 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.802858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.802859 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:36.802905 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.802908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.803585 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.803586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.804158 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.804159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.804531 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.804532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.804581 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.804607 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.804875 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.804883 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.804932 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.804976 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:08:36.805032 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.802562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.0.98.2:35449)
0924 03:08:36.802697 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.802699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.802718 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:08:36.802750 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.802751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.802766 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.802851 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:36.802998 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.803502 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.804256 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.804465 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.804530 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.804712 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.804812 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.804888 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.804937 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.804955 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260924 03:08:36.806221 1662 raft_consensus.cc:1261] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Refusing update from remote peer f33b3a5b02a34b2496f2d8296e141d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.806272 1942 raft_consensus.cc:1261] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Refusing update from remote peer f33b3a5b02a34b2496f2d8296e141d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.806437 2052 consensus_queue.cc:1035] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.806545 2015 consensus_queue.cc:1035] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.807067 2018 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: TxnStatusTablet state changed. Reason: New leader f33b3a5b02a34b2496f2d8296e141d01. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.807156 2018 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.807523 2016 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: TxnStatusTablet state changed. Reason: New leader f33b3a5b02a34b2496f2d8296e141d01. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.807617 2016 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.807780 2052 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.807852 2052 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:36.808301 2016 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.808375 2016 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 98785734dfc742459d93ee283154aee1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.808450 2015 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.808518 2015 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P f33b3a5b02a34b2496f2d8296e141d01: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:36.808943 2018 tablet_replica.cc:439] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33b3a5b02a34b2496f2d8296e141d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 } } peers { permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "0f6df65bdccd45a1a0700372bcc4b998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38779 } } }
I20260924 03:08:36.809018 2018 tablet_replica.cc:442] T 4c19d7dac10346e1bfd9883b9087b386 P 0f6df65bdccd45a1a0700372bcc4b998: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:36.809031 2061 mvcc.cc:205] Tried to move back new op lower bound from 7332738321636446208 to 7332738321517641728. Current Snapshot: MvccSnapshot[applied={T|T < 7332738321636446208}]
I20260924 03:08:36.813802 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.811390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.98.3:53725)
0924 03:08:36.811523 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.811531 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.811609 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.811750 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.811751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.811752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:36.811790 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.811790 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.812595 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.812595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.813127 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.813128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.813506 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.813506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.813545 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.813555 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.813700 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.813702 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.813731 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.813759 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 03:08:36.813802 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.811441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53725 (local address 127.0.98.2:35449)
0924 03:08:36.811477 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.811478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.811589 (+ 111us) server_negotiation.cc:407] Connection header received
0924 03:08:36.811668 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.811668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.811677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.811719 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:36.811846 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.812516 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.813194 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.813442 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.813503 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.813621 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.813650 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.813696 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.813733 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.813750 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:36.815486 1662 raft_consensus.cc:1261] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Refusing update from remote peer 0f6df65bdccd45a1a0700372bcc4b998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.815485 1802 raft_consensus.cc:1261] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Refusing update from remote peer 0f6df65bdccd45a1a0700372bcc4b998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:36.815675 2026 consensus_queue.cc:1035] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33b3a5b02a34b2496f2d8296e141d01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.815749 2018 consensus_queue.cc:1035] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98785734dfc742459d93ee283154aee1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:36.823086 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.820609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39675 (local address 127.0.98.3:38779)
0924 03:08:36.820642 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.820644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.820651 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:36.820688 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.820688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.820697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.820750 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:36.820830 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.821411 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.822366 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.822645 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.822713 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.822891 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.822918 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.822968 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.823010 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.823027 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:36.823088 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.820479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38779 (local address 127.0.98.2:39675)
0924 03:08:36.820606 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:36.820614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:36.820651 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:36.820752 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:36.820753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:36.820754 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:36.820800 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.820801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.821478 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.821478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.822303 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:36.822305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.822745 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.822746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:36.822808 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.822822 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:36.822973 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:36.822975 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:36.823007 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 03:08:36.823035 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260924 03:08:36.834695 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:36.834767 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:36.835289 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7357ca031e14b35857b5372085979f0: Sending DeleteTablet for 3 replicas of tablet 60e66beec5904777be8c5b64d8248a47
I20260924 03:08:36.835471 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e66beec5904777be8c5b64d8248a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45470
I20260924 03:08:36.835496 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e66beec5904777be8c5b64d8248a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54072
I20260924 03:08:36.836035 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e66beec5904777be8c5b64d8248a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57986
I20260924 03:08:36.836081 2076 tablet_replica.cc:330] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: stopping tablet replica
I20260924 03:08:36.836181 2076 raft_consensus.cc:2229] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:36.836184 2077 tablet_replica.cc:330] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: stopping tablet replica
I20260924 03:08:36.836223 2077 raft_consensus.cc:2229] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:36.836267 2076 raft_consensus.cc:2258] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:36.836305 2077 raft_consensus.cc:2258] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:36.836555 2077 ts_tablet_manager.cc:1905] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.836684 2075 tablet_replica.cc:330] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: stopping tablet replica
I20260924 03:08:36.836757 2076 ts_tablet_manager.cc:1905] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.836982 2075 raft_consensus.cc:2229] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:36.837076 2075 raft_consensus.cc:2258] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:36.837423 2075 ts_tablet_manager.cc:1905] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:36.838337 2077 ts_tablet_manager.cc:1918] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:36.838431 2076 ts_tablet_manager.cc:1918] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:36.838455 2077 log.cc:1196] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/60e66beec5904777be8c5b64d8248a47
I20260924 03:08:36.838598 2076 log.cc:1196] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/60e66beec5904777be8c5b64d8248a47
I20260924 03:08:36.838682 2077 ts_tablet_manager.cc:1939] T 60e66beec5904777be8c5b64d8248a47 P f33b3a5b02a34b2496f2d8296e141d01: Deleting consensus metadata
I20260924 03:08:36.838901 2076 ts_tablet_manager.cc:1939] T 60e66beec5904777be8c5b64d8248a47 P 98785734dfc742459d93ee283154aee1: Deleting consensus metadata
I20260924 03:08:36.839020 1492 catalog_manager.cc:4841] TS f33b3a5b02a34b2496f2d8296e141d01 (127.0.98.2:35449): tablet 60e66beec5904777be8c5b64d8248a47 (table test-table [id=be762458f56d49ab93affd6865085a40]) successfully deleted
I20260924 03:08:36.839103 2075 ts_tablet_manager.cc:1918] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:36.839205 1490 catalog_manager.cc:4841] TS 98785734dfc742459d93ee283154aee1 (127.0.98.1:46813): tablet 60e66beec5904777be8c5b64d8248a47 (table test-table [id=be762458f56d49ab93affd6865085a40]) successfully deleted
I20260924 03:08:36.839293 2075 log.cc:1196] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/60e66beec5904777be8c5b64d8248a47
I20260924 03:08:36.839569 2075 ts_tablet_manager.cc:1939] T 60e66beec5904777be8c5b64d8248a47 P 0f6df65bdccd45a1a0700372bcc4b998: Deleting consensus metadata
I20260924 03:08:36.839910 1489 catalog_manager.cc:4841] TS 0f6df65bdccd45a1a0700372bcc4b998 (127.0.98.3:38779): tablet 60e66beec5904777be8c5b64d8248a47 (table test-table [id=be762458f56d49ab93affd6865085a40]) successfully deleted
Sep 24 03:08:36 dist-test-slave-jtnz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:36.853706 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.836931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.0.98.1:46813)
0924 03:08:36.836967 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.836969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.839260 (+ 2291us) server_negotiation.cc:407] Connection header received
0924 03:08:36.839357 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.839358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.839369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.839440 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:36.840091 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.840744 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.841366 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.841625 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.841714 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.850547 (+ 8833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:36.850552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:36.850552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:36.850557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:36.852300 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.852594 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.852597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.852598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.852691 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.852903 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.852904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.852905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.853115 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:36.853247 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.853460 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.853570 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:08:36.853899 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54980
I20260924 03:08:36.860599 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:36.854742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.98.62:45451)
0924 03:08:36.854789 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:36.854791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:36.854800 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:36.854857 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:36.854857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:36.854866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:36.854915 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:36.855311 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.856004 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.856611 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:36.856909 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:36.856977 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:36.858022 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:36.858023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:36.858024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:36.858030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:36.859522 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.859890 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.859891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.859891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.859940 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:36.860169 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:36.860169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:36.860170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:36.860262 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:36.860370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:36.860477 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:08:36.860539 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 03:08:36.860795 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53186
I20260924 03:08:36.861212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1577
I20260924 03:08:36.869421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1717
I20260924 03:08:36.874740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1857
I20260924 03:08:36.879863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 1473
2026-09-24T03:08:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2902 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:36 dist-test-slave-jtnz krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:36 dist-test-slave-jtnz krb5kdc[2084](info): set up 2 sockets
Sep 24 03:08:36 dist-test-slave-jtnz krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260924 03:08:37.837714 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:37 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:37.946264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:38.020780 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:38.020957 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:38.020977 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:38.022132 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:38.022172 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:08:38.022243 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:38.022261 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:38.022284 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:38.022308 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:38.023710 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46869
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:38.023938 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:38.024137 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:38.028226 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:38.028236 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:38.028237 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.028471 2097 server_base.cc:1034] running on GCE node
I20260924 03:08:38.028698 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:38.028966 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:38.030117 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790219318030117 us; error 39 us; skew 500 ppm
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:38.032263 2097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:38.032615 2097 webserver.cc:460] Webserver started at http://127.0.98.62:39065/ using document root <none> and password file <none>
I20260924 03:08:38.032801 2097 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:38.032857 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:38.032990 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:38.034154 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "b75b0217fde24213a7b492660768da1f"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "578b8fa14d1a7e293890d38d3d271e68"
server_key_iv: "8551547054eea7150839c04d6ae6c870"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.034601 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "b75b0217fde24213a7b492660768da1f"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "578b8fa14d1a7e293890d38d3d271e68"
server_key_iv: "8551547054eea7150839c04d6ae6c870"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.035856 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.036460 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.036631 2097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:38.036724 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "b75b0217fde24213a7b492660768da1f"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "578b8fa14d1a7e293890d38d3d271e68"
server_key_iv: "8551547054eea7150839c04d6ae6c870"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.036805 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:38.051896 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:38.053969 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:38.054104 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:38.057937 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46869
I20260924 03:08:38.058017 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46869 every 8 connection(s)
I20260924 03:08:38.058365 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:38.058987 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.059870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2097
I20260924 03:08:38.059969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:38.060143 392 external_mini_cluster.cc:1428] Setting key 7da1a58b6730540312baf9a7170d3442
I20260924 03:08:38.062107 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Bootstrap starting.
I20260924 03:08:38.062681 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.062958 2175 log.cc:824] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:38.063580 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: No bootstrap required, opened a new log
I20260924 03:08:38.064755 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } }
I20260924 03:08:38.064888 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.064926 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75b0217fde24213a7b492660768da1f, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.065040 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } }
I20260924 03:08:38.065124 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:38.065163 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:38.065227 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.065685 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } }
I20260924 03:08:38.065807 2175 leader_election.cc:302] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75b0217fde24213a7b492660768da1f; no voters:
I20260924 03:08:38.065973 2175 leader_election.cc:288] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:38.066018 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:38.066172 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [term 1 LEADER]: Becoming Leader. State: Replica: b75b0217fde24213a7b492660768da1f, State: Running, Role: LEADER
I20260924 03:08:38.066233 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:38.066294 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } }
I20260924 03:08:38.066642 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75b0217fde24213a7b492660768da1f. Latest consensus state: current_term: 1 leader_uuid: "b75b0217fde24213a7b492660768da1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } } }
I20260924 03:08:38.066725 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:38.066651 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75b0217fde24213a7b492660768da1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b0217fde24213a7b492660768da1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46869 } } }
I20260924 03:08:38.066864 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f [sys.catalog]: This master's current role is: LEADER
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.067101 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:38.067605 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:38.069429 2188 catalog_manager.cc:1347] Generated new cluster ID: cabd3fb6acde44b586556154b57008bd
I20260924 03:08:38.069480 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:38.070868 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.060607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.98.62:46869)
0924 03:08:38.060755 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.060770 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.060793 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:08:38.061174 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.061180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.061223 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.061323 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.062264 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.063249 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.064028 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.064780 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.064892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.067390 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.067394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.067394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.067405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.069319 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.069708 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.069711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.069711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.069801 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.069996 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.069996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.069997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.070274 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.070432 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.070507 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.070632 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 03:08:38.075189 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:38.075661 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:38.080873 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Generated new TSK 0
I20260924 03:08:38.081015 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:38.110885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46869
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:38.206610 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:38.206828 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:38.206874 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:38.207937 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:38.207996 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:38.208030 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:08:38.208057 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:38.208096 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:38.209381 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:38.209614 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:38.209831 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:38.215109 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:38.215179 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.215242 2201 server_base.cc:1034] running on GCE node
W20260924 03:08:38.215121 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.215593 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:38.215837 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:38.216979 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790219318216968 us; error 29 us; skew 500 ppm
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:38.219323 2201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:38.219811 2201 webserver.cc:460] Webserver started at http://127.0.98.1:42447/ using document root <none> and password file <none>
I20260924 03:08:38.220026 2201 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:38.220089 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:38.220233 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:38.221102 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "188e1648da9e43009f170d38e3099640"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "1730847c358db7d77627b22b93a3a287"
server_key_iv: "902a57088ae007873abbd43598662f9e"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.221416 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "188e1648da9e43009f170d38e3099640"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "1730847c358db7d77627b22b93a3a287"
server_key_iv: "902a57088ae007873abbd43598662f9e"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.222683 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:38.223316 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.223575 2201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:38.223666 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "188e1648da9e43009f170d38e3099640"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "1730847c358db7d77627b22b93a3a287"
server_key_iv: "902a57088ae007873abbd43598662f9e"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.223744 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:38.241132 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:38.243472 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:38.243625 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:38.244107 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:38.244161 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.244205 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:38.244226 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.251305 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37311
I20260924 03:08:38.251621 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:38.251673 2331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37311 every 8 connection(s)
I20260924 03:08:38.253363 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.244466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38381 (local address 127.0.98.62:46869)
0924 03:08:38.244510 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.244512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.244783 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:08:38.245352 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.245353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.245362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.245402 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.246234 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.247042 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.248059 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.248395 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.248466 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.250878 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.250879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.250879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.250886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.252305 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.252640 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.252641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.252641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.252687 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.252876 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.252876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.252876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.252964 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.253010 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.253247 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.253305 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:38.253520 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.244472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.1:38381)
0924 03:08:38.244644 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.244689 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.245259 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.245467 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.245471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.245834 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.246093 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.246099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.247132 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.247133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.247960 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.247962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.248505 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.248506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.248640 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.249099 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.249115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.250802 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.252373 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.252376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.252378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.252577 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.252741 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.252742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.252742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.252825 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.253015 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.253018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.253134 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.253211 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.253326 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260924 03:08:38.253968 2332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46869
I20260924 03:08:38.254045 2332 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:38.254247 2332 heartbeater.cc:511] Master 127.0.98.62:46869 requested a full tablet report, sending...
I20260924 03:08:38.254799 2129 ts_manager.cc:194] Registered new tserver with Master: 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311)
I20260924 03:08:38.255661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2201
I20260924 03:08:38.255697 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38381
I20260924 03:08:38.255759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:38.255928 392 external_mini_cluster.cc:1428] Setting key 3d1aae561fa79dfd5c0d9801b98988ad
I20260924 03:08:38.260561 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.255028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43275 (local address 127.0.98.62:46869)
0924 03:08:38.255065 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.255066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.255182 (+ 116us) server_negotiation.cc:407] Connection header received
0924 03:08:38.255239 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.255239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.255247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.255288 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.255612 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.256251 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.256971 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.257167 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.257246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.258252 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.258253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.258254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.258262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.259530 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.259917 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.259918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.259918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.259960 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.260112 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.260112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.260112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.260187 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.260260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.260420 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.260484 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:08:38.260587 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.254975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.1:43275)
0924 03:08:38.255117 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.255126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.255188 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.255300 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.255300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.255470 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.255530 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.255531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.256365 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.256366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.256883 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.256886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.257249 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.257250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.257322 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.257769 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.257774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.258164 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.259611 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.259612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.259613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.259826 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.260001 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.260001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.260002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.260062 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.260266 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.260267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.260308 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.260382 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.260478 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:38.278775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46869
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:38.355305 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:38.355484 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:38.355517 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:38.356518 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:38.356549 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:38.356566 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:08:38.356577 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:38.356592 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:38.357818 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:38.357973 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:38.358156 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:38.362789 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:38.362767 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.362902 2341 server_base.cc:1034] running on GCE node
W20260924 03:08:38.362780 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.363234 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:38.363464 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:38.364598 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790219318364586 us; error 28 us; skew 500 ppm
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:38.366883 2341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:38.367249 2341 webserver.cc:460] Webserver started at http://127.0.98.2:43341/ using document root <none> and password file <none>
I20260924 03:08:38.367457 2341 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:38.367522 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:38.367666 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:38.368575 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "8003d51534d748d39c2182de212ebaab"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "36c468f5d7dfb05612f7d43e2a0e1a98"
server_key_iv: "780053c99318950b1f88927f2dc80e7f"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.368930 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8003d51534d748d39c2182de212ebaab"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "36c468f5d7dfb05612f7d43e2a0e1a98"
server_key_iv: "780053c99318950b1f88927f2dc80e7f"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.370123 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:38.370816 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.370967 2341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:38.371064 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "8003d51534d748d39c2182de212ebaab"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "36c468f5d7dfb05612f7d43e2a0e1a98"
server_key_iv: "780053c99318950b1f88927f2dc80e7f"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.371141 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:38.402243 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:38.403998 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:38.404155 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:38.404601 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:38.404661 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.404702 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:38.404747 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.411265 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39481
I20260924 03:08:38.411348 2471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39481 every 8 connection(s)
I20260924 03:08:38.411610 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:38.413162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2341
I20260924 03:08:38.413245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:38.413401 392 external_mini_cluster.cc:1428] Setting key 1cee42dffdf59a7c38ddfe14002430b2
I20260924 03:08:38.414403 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.405076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57335 (local address 127.0.98.62:46869)
0924 03:08:38.405141 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.405143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.405348 (+ 205us) server_negotiation.cc:407] Connection header received
0924 03:08:38.405797 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.405798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.405811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.405864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.406605 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.407426 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.408950 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.409276 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.409354 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.411677 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.411679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.411679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.411687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.413073 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.413498 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.413500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.413500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.413557 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.413813 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.413814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.413814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.413935 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.413997 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.414292 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.414348 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:08:38.414669 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.405077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.2:57335)
0924 03:08:38.405241 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.405270 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.405702 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.405922 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.405925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.406242 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.406512 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.406518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.407545 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.407546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.408855 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.408858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.409368 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.409369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.409479 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.409943 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.409952 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.411576 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.413149 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.413154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.413156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.413415 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.413628 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.413629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.413629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.413752 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.414000 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.414005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.414149 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.414271 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.414419 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260924 03:08:38.415606 2472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46869
I20260924 03:08:38.415693 2472 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:38.415853 2472 heartbeater.cc:511] Master 127.0.98.62:46869 requested a full tablet report, sending...
I20260924 03:08:38.416165 2129 ts_manager.cc:194] Registered new tserver with Master: 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:38.416591 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57335
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:08:38.422287 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.416518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.2:44073)
0924 03:08:38.416699 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.416707 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.416770 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.416871 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.416871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.417036 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.417082 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.417092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.417997 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.417997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.418510 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.418512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.419009 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.419010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.419059 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.419454 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.419457 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.419691 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.421285 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.421285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.421286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.421435 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.421636 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.421637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.421637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.421684 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.422017 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.422017 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.422047 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.422096 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.422159 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260924 03:08:38.422328 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.416594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44073 (local address 127.0.98.62:46869)
0924 03:08:38.416648 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.416650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.416751 (+ 101us) server_negotiation.cc:407] Connection header received
0924 03:08:38.416813 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.416814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.416823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.416866 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.417167 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.417911 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.418590 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.418936 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.419014 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.419769 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.419770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.419770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.419777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.421212 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.421501 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.421503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.421503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.421567 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.421740 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.421741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.421741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.421922 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.422013 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.422121 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.422222 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:38.437950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46869
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:38.512946 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:38.513151 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:38.513191 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:38.514279 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:38.514329 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:38.514359 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:08:38.514386 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:08:38.514420 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:38.515810 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:38.516054 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:38.516291 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:38.520896 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:38.520949 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.521070 2481 server_base.cc:1034] running on GCE node
W20260924 03:08:38.520901 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:38.521322 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:38.521526 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:38.522672 2481 hybrid_clock.cc:648] HybridClock initialized: now 1790219318522664 us; error 27 us; skew 500 ppm
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:38.524983 2481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:38.525444 2481 webserver.cc:460] Webserver started at http://127.0.98.3:34461/ using document root <none> and password file <none>
I20260924 03:08:38.525612 2481 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:38.525645 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:38.525717 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:38.526767 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "af9ac651c0fd41b1ad1bfec96bc30e91"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "61856778064a463e352205639b6bf75c"
server_key_iv: "fb518d14c5bc14818575edc6ef46b0c2"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.527096 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af9ac651c0fd41b1ad1bfec96bc30e91"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "61856778064a463e352205639b6bf75c"
server_key_iv: "fb518d14c5bc14818575edc6ef46b0c2"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.528435 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:38.529404 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.529590 2481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.529680 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "af9ac651c0fd41b1ad1bfec96bc30e91"
format_stamp: "Formatted at 2026-09-24 03:08:38 on dist-test-slave-jtnz"
server_key: "61856778064a463e352205639b6bf75c"
server_key_iv: "fb518d14c5bc14818575edc6ef46b0c2"
server_key_version: "encryptionkey@0"
I20260924 03:08:38.529764 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:38.555966 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:38.557974 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:38.558151 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:38.558703 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:38.558763 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.558794 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:38.558821 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.565562 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32843
I20260924 03:08:38.565591 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32843 every 8 connection(s)
I20260924 03:08:38.565918 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.569042 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.559219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55895 (local address 127.0.98.62:46869)
0924 03:08:38.559290 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.559292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.559445 (+ 153us) server_negotiation.cc:407] Connection header received
0924 03:08:38.559901 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.559902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.559913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.559999 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.560766 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.561568 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.562747 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.563096 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.563175 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.566537 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.566538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.566539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.566544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.568071 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.568391 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.568391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.568392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.568437 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.568613 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.568614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.568614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.568710 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.568760 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.568941 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.568988 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:08:38.569269 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.559181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.3:55895)
0924 03:08:38.559321 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.559358 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.559803 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.560086 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.560089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.560427 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.560670 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.560676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.561663 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.561664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.562683 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.562685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.563265 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.563266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.563426 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.564548 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.564557 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.566441 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.568135 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.568140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.568142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.568332 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.568477 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.568477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.568478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.568561 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.568758 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.568760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.568867 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.568932 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.569043 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260924 03:08:38.569734 2612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46869
I20260924 03:08:38.569813 2612 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:38.569968 2612 heartbeater.cc:511] Master 127.0.98.62:46869 requested a full tablet report, sending...
I20260924 03:08:38.570310 2129 ts_manager.cc:194] Registered new tserver with Master: af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843)
I20260924 03:08:38.570683 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55895
I20260924 03:08:38.572057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2481
I20260924 03:08:38.572136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:38.572288 392 external_mini_cluster.cc:1428] Setting key 4baf4d522c606c141f082f49b141dd76
I20260924 03:08:38.573113 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:38.576059 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.570734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44109 (local address 127.0.98.62:46869)
0924 03:08:38.570767 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.570769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.571098 (+ 329us) server_negotiation.cc:407] Connection header received
0924 03:08:38.571135 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.571135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.571143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.571186 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.571495 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.572038 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.572667 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.572897 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.572941 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.573940 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.573941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.573941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.573947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.575130 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.575479 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.575479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.575480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.575521 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.575693 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.575694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.575694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.575769 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.575811 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.575954 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.576013 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:38.576084 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.570676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.98.3:44109)
0924 03:08:38.571010 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.571021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.571088 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.571191 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.571191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.571364 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.571426 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.571427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.572122 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.572123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.572583 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.572585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.573006 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.573007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.573068 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.573484 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.573488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.573857 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.575197 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.575198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.575199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.575405 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.575577 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.575578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.575579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.575639 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.575825 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.575826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.575867 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.575940 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.576023 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":73,"threads_started":1}
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.594270 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.588023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.0.98.62:46869)
0924 03:08:38.588078 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.588079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.588088 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:38.588132 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.588133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.588141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.588210 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.588751 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.589178 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.589882 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.590059 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.590143 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.592096 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.592098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.592098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.592105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.593166 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.593502 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.593503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.593503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.593557 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.593797 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.593798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.593798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.593895 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.593992 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.594084 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.594153 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:08:38.595562 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:38.596983 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:38.602746 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.0.1:51982)
0924 03:08:38.599272 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.599280 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.599337 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.599713 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.599713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.599715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.599763 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.599764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600918 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.600919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.601592 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.601593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.602452 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.602453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.602537 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.602550 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.602620 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.602667 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260924 03:08:38.602766 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.98.1:37311)
0924 03:08:38.599558 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.599562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.599568 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:08:38.599621 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.599623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.599637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.599707 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.599839 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600827 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.601679 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.602357 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.602449 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.602507 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.602639 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.602674 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":63,"threads_started":1}
I20260924 03:08:38.603112 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32843 (local address 127.0.0.1:41302)
0924 03:08:38.599394 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.599403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.599453 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.599648 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.599648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.599650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.599701 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.599703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600884 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.600884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.601915 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.601917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.602866 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.602867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.602955 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.602959 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.602987 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.603024 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":34,"thread_start_us":65,"threads_started":1}
I20260924 03:08:38.603380 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39481 (local address 127.0.0.1:38346)
0924 03:08:38.599241 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.599277 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.599337 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.599629 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.599632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.599634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.599698 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.599700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600642 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.600643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.601831 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.601833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.603154 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.603155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.603215 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.603219 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.603267 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.603307 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":50,"thread_start_us":51,"threads_started":1}
I20260924 03:08:38.603435 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.98.3:32843)
0924 03:08:38.599503 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.599506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.599514 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:38.599542 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.599545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.599561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.599767 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.599816 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600806 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.602007 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.602755 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.603222 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.603285 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.603311 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.603347 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 03:08:38.603451 2266 tablet_service.cc:1467] Processing CreateTablet for tablet c3e99bcb475e4291a080b30b78be5061 (DEFAULT_TABLE table=test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.603727 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.599111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.0.98.2:39481)
0924 03:08:38.599460 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.599465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.599474 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:38.599511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.599513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.599528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.599615 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.599793 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.600563 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.602020 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.602755 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.603137 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.603340 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.603363 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.603402 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":69,"threads_started":1}
I20260924 03:08:38.603842 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e99bcb475e4291a080b30b78be5061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.606725 2635 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Bootstrap starting.
I20260924 03:08:38.606853 2406 tablet_service.cc:1467] Processing CreateTablet for tablet c3e99bcb475e4291a080b30b78be5061 (DEFAULT_TABLE table=test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.606846 2546 tablet_service.cc:1467] Processing CreateTablet for tablet c3e99bcb475e4291a080b30b78be5061 (DEFAULT_TABLE table=test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.607155 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e99bcb475e4291a080b30b78be5061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.607164 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e99bcb475e4291a080b30b78be5061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.607334 2635 tablet_bootstrap.cc:654] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.607622 2635 log.cc:824] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:38.608270 2635 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: No bootstrap required, opened a new log
I20260924 03:08:38.608347 2635 ts_tablet_manager.cc:1397] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:38.609727 2635 raft_consensus.cc:348] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.609872 2635 raft_consensus.cc:374] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.609913 2635 raft_consensus.cc:729] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188e1648da9e43009f170d38e3099640, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.610020 2635 consensus_queue.cc:260] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.610361 2635 ts_tablet_manager.cc:1428] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:08:38.610361 2637 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Bootstrap starting.
I20260924 03:08:38.610491 2332 heartbeater.cc:503] Master 127.0.98.62:46869 was elected leader, sending a full tablet report...
I20260924 03:08:38.611023 2639 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Bootstrap starting.
I20260924 03:08:38.611172 2637 tablet_bootstrap.cc:654] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.611517 2637 log.cc:824] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:38.611765 2639 tablet_bootstrap.cc:654] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.612043 2639 log.cc:824] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:38.612164 2637 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: No bootstrap required, opened a new log
I20260924 03:08:38.612224 2637 ts_tablet_manager.cc:1397] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:38.612671 2639 tablet_bootstrap.cc:492] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: No bootstrap required, opened a new log
I20260924 03:08:38.612720 2639 ts_tablet_manager.cc:1397] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:38.613548 2637 raft_consensus.cc:348] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.613679 2637 raft_consensus.cc:374] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.613722 2637 raft_consensus.cc:729] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.613824 2637 consensus_queue.cc:260] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.614084 2637 ts_tablet_manager.cc:1428] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:08:38.614105 2612 heartbeater.cc:503] Master 127.0.98.62:46869 was elected leader, sending a full tablet report...
I20260924 03:08:38.614220 2639 raft_consensus.cc:348] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.614375 2639 raft_consensus.cc:374] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.614437 2639 raft_consensus.cc:729] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.614540 2639 consensus_queue.cc:260] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.614815 2639 ts_tablet_manager.cc:1428] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:08:38.614851 2472 heartbeater.cc:503] Master 127.0.98.62:46869 was elected leader, sending a full tablet report...
I20260924 03:08:38.658849 2643 raft_consensus.cc:482] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:38.658967 2643 raft_consensus.cc:504] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.659327 2643 leader_election.cc:288] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843), 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311)
W20260924 03:08:38.662137 2473 tablet.cc:2369] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:38.662261 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.659546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.98.2:34893)
0924 03:08:38.659628 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.659640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.659681 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.659974 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.659975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.659976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.660039 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.660041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.660713 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.660715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.661426 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.661428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.661997 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.661998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.662069 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.662072 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.662138 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.662178 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:08:38.662367 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.659762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34893 (local address 127.0.98.1:37311)
0924 03:08:38.659811 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.659813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.659821 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:38.659863 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.659864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.659883 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.659943 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.660137 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.660631 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.661515 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.661896 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.661988 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.661999 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.662157 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.662297 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:08:38.662514 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.659645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56707 (local address 127.0.98.3:32843)
0924 03:08:38.659687 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.659689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.660080 (+ 391us) server_negotiation.cc:407] Connection header received
0924 03:08:38.660152 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.660155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.660169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.660234 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.660380 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.660820 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.661704 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.662105 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.662171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.662200 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.662389 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.662437 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:38.662618 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e99bcb475e4291a080b30b78be5061" candidate_uuid: "8003d51534d748d39c2182de212ebaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640" is_pre_election: true
I20260924 03:08:38.662758 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.659546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32843 (local address 127.0.98.2:56707)
0924 03:08:38.659992 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.660002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.660084 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.660238 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.660238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.660240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.660311 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.660312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.660901 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.660902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.661617 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.661619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.662233 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.662234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.662305 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.662308 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.662622 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.662660 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":265,"thread_start_us":126,"threads_started":1}
I20260924 03:08:38.662784 2286 raft_consensus.cc:2454] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8003d51534d748d39c2182de212ebaab in term 0.
I20260924 03:08:38.663022 2360 leader_election.cc:302] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188e1648da9e43009f170d38e3099640, 8003d51534d748d39c2182de212ebaab; no voters:
I20260924 03:08:38.663053 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e99bcb475e4291a080b30b78be5061" candidate_uuid: "8003d51534d748d39c2182de212ebaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" is_pre_election: true
I20260924 03:08:38.663143 2643 raft_consensus.cc:2781] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:38.663203 2643 raft_consensus.cc:482] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:38.663249 2643 raft_consensus.cc:3037] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.663244 2566 raft_consensus.cc:2454] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8003d51534d748d39c2182de212ebaab in term 0.
I20260924 03:08:38.664062 2643 raft_consensus.cc:504] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.664186 2643 leader_election.cc:288] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [CANDIDATE]: Term 1 election: Requested vote from peers af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843), 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311)
I20260924 03:08:38.664328 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e99bcb475e4291a080b30b78be5061" candidate_uuid: "8003d51534d748d39c2182de212ebaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91"
I20260924 03:08:38.664379 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e99bcb475e4291a080b30b78be5061" candidate_uuid: "8003d51534d748d39c2182de212ebaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640"
I20260924 03:08:38.664431 2566 raft_consensus.cc:3037] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.664475 2286 raft_consensus.cc:3037] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.665067 2566 raft_consensus.cc:2454] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8003d51534d748d39c2182de212ebaab in term 1.
I20260924 03:08:38.665154 2286 raft_consensus.cc:2454] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8003d51534d748d39c2182de212ebaab in term 1.
I20260924 03:08:38.665264 2357 leader_election.cc:302] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:38.665354 2643 raft_consensus.cc:2781] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:38.665480 2643 raft_consensus.cc:686] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 1 LEADER]: Becoming Leader. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Running, Role: LEADER
I20260924 03:08:38.665572 2643 consensus_queue.cc:237] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } }
I20260924 03:08:38.666285 2129 catalog_manager.cc:5495] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8003d51534d748d39c2182de212ebaab (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8003d51534d748d39c2182de212ebaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:08:38 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:38.705051 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.698712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46869 (local address 127.0.0.1:39080)
0924 03:08:38.698894 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.698904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.698986 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.699140 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.699141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.699420 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:38.699463 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.699465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.700099 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.700100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.700593 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.700594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.700929 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.700930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.700978 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.701418 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:38.701422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:38.702724 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:38.704040 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.704046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.704048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.704230 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.704420 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:38.704421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:38.704421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:38.704477 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:38.704720 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:38.704726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:38.704763 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.704837 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.704941 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260924 03:08:38.705056 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.698780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.98.62:46869)
0924 03:08:38.698862 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.698865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.698969 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:08:38.699056 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.699057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.699066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.699135 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:38.699561 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.700006 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.700679 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.700850 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.700928 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.702809 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:38.702811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:38.702811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:38.702818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:38.703950 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.704301 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.704303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.704303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.704358 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:38.704555 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:38.704556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:38.704556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:38.704652 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:38.704715 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.704857 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.704986 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":57}
I20260924 03:08:38.706709 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:08:38.706889 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:38.708598 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 03aef7e054934b509072076d73630aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e53e8695a27a4e2da5aab34a3f13123a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:38.708655 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 03aef7e054934b509072076d73630aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e53e8695a27a4e2da5aab34a3f13123a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:38.708783 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03aef7e054934b509072076d73630aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.708801 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03aef7e054934b509072076d73630aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.710090 2637 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: Bootstrap starting.
I20260924 03:08:38.710145 2635 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: Bootstrap starting.
I20260924 03:08:38.710184 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 03aef7e054934b509072076d73630aed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e53e8695a27a4e2da5aab34a3f13123a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:08:38.710331 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03aef7e054934b509072076d73630aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.710733 2637 tablet_bootstrap.cc:654] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.710821 2635 tablet_bootstrap.cc:654] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.711356 2639 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: Bootstrap starting.
I20260924 03:08:38.711418 2637 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: No bootstrap required, opened a new log
I20260924 03:08:38.711462 2637 ts_tablet_manager.cc:1397] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.711498 2635 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: No bootstrap required, opened a new log
I20260924 03:08:38.711535 2635 ts_tablet_manager.cc:1397] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.711617 2637 raft_consensus.cc:348] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.711678 2637 raft_consensus.cc:374] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.711699 2637 raft_consensus.cc:729] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.711664 2635 raft_consensus.cc:348] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.711725 2635 raft_consensus.cc:374] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.711719 2639 tablet_bootstrap.cc:654] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.711747 2635 raft_consensus.cc:729] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188e1648da9e43009f170d38e3099640, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.711750 2637 consensus_queue.cc:260] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.711786 2635 consensus_queue.cc:260] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.711839 2637 ts_tablet_manager.cc:1428] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.711867 2635 ts_tablet_manager.cc:1428] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.711951 2642 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.712024 2642 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.712009 2638 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.712185 2638 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.712544 2639 tablet_bootstrap.cc:492] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: No bootstrap required, opened a new log
I20260924 03:08:38.712596 2639 ts_tablet_manager.cc:1397] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.712731 2639 raft_consensus.cc:348] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.712817 2639 raft_consensus.cc:374] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.712855 2639 raft_consensus.cc:729] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.712910 2639 consensus_queue.cc:260] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.713126 2639 ts_tablet_manager.cc:1428] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.713186 2643 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.713279 2643 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 03:08:38.752332 2333 tablet.cc:2369] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:38.763234 2638 raft_consensus.cc:482] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:38.763288 2638 raft_consensus.cc:504] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.763538 2638 leader_election.cc:288] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481), af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843)
I20260924 03:08:38.766347 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.763910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32843 (local address 127.0.98.1:57521)
0924 03:08:38.763984 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.763993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.764044 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.764170 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.764170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.764172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.764252 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.764254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.764815 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.764815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.765570 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.765572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.766076 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.766077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.766142 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.766147 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.766227 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.766266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":25,"thread_start_us":46,"threads_started":1}
I20260924 03:08:38.766350 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.763965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57521 (local address 127.0.98.3:32843)
0924 03:08:38.764004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.764006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.764036 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:08:38.764090 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.764091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.764098 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.764158 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.764317 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.764732 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.765662 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.766002 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.766052 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.766060 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.766239 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.766277 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:08:38.766564 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.763788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53909 (local address 127.0.98.2:39481)
0924 03:08:38.763849 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.763851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.763913 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:08:38.764003 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.764004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.764015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.764043 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:38.764328 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.764724 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.765669 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.766129 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.766258 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.766267 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.766463 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.766512 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:08:38.766574 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.763714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39481 (local address 127.0.98.1:53909)
0924 03:08:38.763845 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.763856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.763922 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.764118 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.764119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.764121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:38.764246 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.764250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.764792 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.764793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.765584 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.765587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.766278 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.766279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.766358 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.766362 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.766449 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.766488 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":45,"thread_start_us":68,"threads_started":1}
I20260924 03:08:38.766994 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03aef7e054934b509072076d73630aed" candidate_uuid: "188e1648da9e43009f170d38e3099640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" is_pre_election: true
I20260924 03:08:38.767076 2566 raft_consensus.cc:2454] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188e1648da9e43009f170d38e3099640 in term 0.
I20260924 03:08:38.767019 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03aef7e054934b509072076d73630aed" candidate_uuid: "188e1648da9e43009f170d38e3099640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab" is_pre_election: true
I20260924 03:08:38.767170 2426 raft_consensus.cc:2454] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188e1648da9e43009f170d38e3099640 in term 0.
I20260924 03:08:38.767220 2217 leader_election.cc:302] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188e1648da9e43009f170d38e3099640, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:38.767345 2638 raft_consensus.cc:2781] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:38.767395 2638 raft_consensus.cc:482] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:38.767409 2638 raft_consensus.cc:3037] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.768108 2638 raft_consensus.cc:504] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.768251 2638 leader_election.cc:288] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [CANDIDATE]: Term 1 election: Requested vote from peers 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481), af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843)
I20260924 03:08:38.768404 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03aef7e054934b509072076d73630aed" candidate_uuid: "188e1648da9e43009f170d38e3099640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab"
I20260924 03:08:38.768496 2426 raft_consensus.cc:3037] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.768534 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03aef7e054934b509072076d73630aed" candidate_uuid: "188e1648da9e43009f170d38e3099640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91"
I20260924 03:08:38.768604 2566 raft_consensus.cc:3037] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:38.769372 2566 raft_consensus.cc:2454] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188e1648da9e43009f170d38e3099640 in term 1.
I20260924 03:08:38.769380 2426 raft_consensus.cc:2454] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188e1648da9e43009f170d38e3099640 in term 1.
I20260924 03:08:38.769516 2217 leader_election.cc:302] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188e1648da9e43009f170d38e3099640, 8003d51534d748d39c2182de212ebaab; no voters:
I20260924 03:08:38.769596 2638 raft_consensus.cc:2781] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:38.769706 2638 raft_consensus.cc:686] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [term 1 LEADER]: Becoming Leader. State: Replica: 188e1648da9e43009f170d38e3099640, State: Running, Role: LEADER
I20260924 03:08:38.769810 2638 consensus_queue.cc:237] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.770224 2658 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: TxnStatusTablet state changed. Reason: New leader 188e1648da9e43009f170d38e3099640. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.770339 2658 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:38.770507 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:08:38.770632 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:08:38.770553 2128 catalog_manager.cc:5495] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 188e1648da9e43009f170d38e3099640 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20260924 03:08:38.786006 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.783919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.0.98.1:37311)
0924 03:08:38.783958 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.783960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.783984 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:08:38.784066 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.784067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.784081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.784155 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:38.784256 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.784662 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.785322 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.785492 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.785559 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.785734 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.785815 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.785884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.785933 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.785952 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:08:38.786031 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.783879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.0.1:51986)
0924 03:08:38.783946 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.783956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.784001 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.784158 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.784159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.784160 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:38.784201 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.784203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.784736 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.784737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.785244 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.785246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.785558 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.785559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.785614 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.785628 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.785876 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.785884 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.785928 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.785965 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:08:38.787596 2566 raft_consensus.cc:1261] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Refusing update from remote peer 188e1648da9e43009f170d38e3099640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:38.787608 2426 raft_consensus.cc:1261] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Refusing update from remote peer 188e1648da9e43009f170d38e3099640: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:38.787914 2638 consensus_queue.cc:1035] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:38.788033 2658 consensus_queue.cc:1035] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:38.789103 2643 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: TxnStatusTablet state changed. Reason: New leader 188e1648da9e43009f170d38e3099640. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.789114 2642 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: TxnStatusTablet state changed. Reason: New leader 188e1648da9e43009f170d38e3099640. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.789178 2643 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.789186 2642 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.790166 2658 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.790257 2658 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:38.790373 2663 mvcc.cc:205] Tried to move back new op lower bound from 7332738329751908352 to 7332738329681825792. Current Snapshot: MvccSnapshot[applied={T|T < 7332738329751908352}]
I20260924 03:08:38.790843 2642 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.790922 2642 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P af9ac651c0fd41b1ad1bfec96bc30e91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.791096 2638 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.791165 2638 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 188e1648da9e43009f170d38e3099640: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:08:38.791735 2643 tablet_replica.cc:439] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "188e1648da9e43009f170d38e3099640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } } }
I20260924 03:08:38.791813 2643 tablet_replica.cc:442] T 03aef7e054934b509072076d73630aed P 8003d51534d748d39c2182de212ebaab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:08:38.794546 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.792506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.0.98.2:39481)
0924 03:08:38.792554 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.792555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.792562 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:38.792586 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.792587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.792600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.792668 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:38.792822 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.793197 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.793844 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.794023 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.794120 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.794277 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.794372 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.794448 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.794467 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.794484 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:08:38.799099 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.796674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57165 (local address 127.0.98.1:37311)
0924 03:08:38.796719 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.796721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.796728 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:38.796775 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.796776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.796784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.796824 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:38.796917 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.797279 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.798365 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.798554 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.798640 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.798826 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.798861 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.798923 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.799002 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.799020 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:08:38.799125 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.796612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.98.2:57165)
0924 03:08:38.796671 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.796679 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.796726 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.796826 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.796827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.796827 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:38.796872 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.796873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.797350 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.797351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.798250 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.798253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.798643 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.798645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.798705 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.798725 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.798924 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.798926 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.798998 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.799047 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:08:38.800983 2566 raft_consensus.cc:1261] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Refusing update from remote peer 8003d51534d748d39c2182de212ebaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:38.801036 2286 raft_consensus.cc:1261] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Refusing update from remote peer 8003d51534d748d39c2182de212ebaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:38.801270 2643 consensus_queue.cc:1035] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:38.801380 2645 consensus_queue.cc:1035] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:38.802688 2665 mvcc.cc:205] Tried to move back new op lower bound from 7332738329807425536 to 7332738329254658048. Current Snapshot: MvccSnapshot[applied={T|T < 7332738329807425536}]
I20260924 03:08:38.803539 2664 mvcc.cc:205] Tried to move back new op lower bound from 7332738329807425536 to 7332738329254658048. Current Snapshot: MvccSnapshot[applied={T|T < 7332738329807425536}]
I20260924 03:08:38.809875 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.807333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37607 (local address 127.0.98.2:39481)
0924 03:08:38.807362 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:38.807363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:38.807623 (+ 260us) server_negotiation.cc:407] Connection header received
0924 03:08:38.807678 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:38.807678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:38.807686 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:38.807730 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:38.807832 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.808239 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.809321 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.809481 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.809530 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.809721 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.809740 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.809769 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:38.809827 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:08:38.809838 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 03:08:38.809940 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:38.807281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39481 (local address 127.0.98.1:37607)
0924 03:08:38.807570 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:38.807580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:38.807634 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:38.807734 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:38.807735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:38.807736 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:38.807781 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.807782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.808329 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.808330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.809239 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:38.809242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:38.809571 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:38.809571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:38.809639 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:38.809652 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:38.809782 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:38.809783 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:38.809825 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:08:38.809858 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":66,"threads_started":1}
I20260924 03:08:38.816277 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:38.816386 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:38.816970 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Sending DeleteTablet for 3 replicas of tablet c3e99bcb475e4291a080b30b78be5061
I20260924 03:08:38.817190 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet c3e99bcb475e4291a080b30b78be5061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38346
I20260924 03:08:38.817305 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet c3e99bcb475e4291a080b30b78be5061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41302
I20260924 03:08:38.817384 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet c3e99bcb475e4291a080b30b78be5061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51982
I20260924 03:08:38.817616 2684 tablet_replica.cc:330] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: stopping tablet replica
I20260924 03:08:38.817612 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:38.817776 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:38.818027 2684 raft_consensus.cc:2229] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:38.819002 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 4d362ce7122348c3bb710007040a8b5a (DEFAULT_TABLE table=test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.819025 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 4d362ce7122348c3bb710007040a8b5a (DEFAULT_TABLE table=test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.819113 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d362ce7122348c3bb710007040a8b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.819149 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d362ce7122348c3bb710007040a8b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.820046 2637 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Bootstrap starting.
I20260924 03:08:38.818919 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 4d362ce7122348c3bb710007040a8b5a (DEFAULT_TABLE table=test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:38.820246 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d362ce7122348c3bb710007040a8b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:38.820308 2639 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Bootstrap starting.
I20260924 03:08:38.820431 2637 tablet_bootstrap.cc:654] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.820704 2639 tablet_bootstrap.cc:654] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.821282 2639 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: No bootstrap required, opened a new log
I20260924 03:08:38.821331 2639 ts_tablet_manager.cc:1397] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:38.821465 2639 raft_consensus.cc:348] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.821524 2639 raft_consensus.cc:374] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.821542 2639 raft_consensus.cc:729] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.821553 2635 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Bootstrap starting.
I20260924 03:08:38.821589 2639 consensus_queue.cc:260] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.821663 2684 raft_consensus.cc:2258] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:38.821674 2639 ts_tablet_manager.cc:1428] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.821818 2683 tablet_replica.cc:330] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: stopping tablet replica
I20260924 03:08:38.821893 2683 raft_consensus.cc:2229] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:38.821936 2635 tablet_bootstrap.cc:654] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:38.821964 2683 raft_consensus.cc:2258] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:38.822124 2684 ts_tablet_manager.cc:1905] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:38.822435 2683 ts_tablet_manager.cc:1905] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:38.823896 2684 ts_tablet_manager.cc:1918] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:38.823889 2683 ts_tablet_manager.cc:1918] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:38.823957 2683 log.cc:1196] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/c3e99bcb475e4291a080b30b78be5061
I20260924 03:08:38.824070 2684 log.cc:1196] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/c3e99bcb475e4291a080b30b78be5061
I20260924 03:08:38.824203 2683 ts_tablet_manager.cc:1939] T c3e99bcb475e4291a080b30b78be5061 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting consensus metadata
I20260924 03:08:38.824457 2684 ts_tablet_manager.cc:1939] T c3e99bcb475e4291a080b30b78be5061 P 188e1648da9e43009f170d38e3099640: Deleting consensus metadata
I20260924 03:08:38.824594 2113 catalog_manager.cc:4841] TS af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843): tablet c3e99bcb475e4291a080b30b78be5061 (table test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]) successfully deleted
I20260924 03:08:38.824934 2116 catalog_manager.cc:4841] TS 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311): tablet c3e99bcb475e4291a080b30b78be5061 (table test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]) successfully deleted
I20260924 03:08:38.829213 2682 tablet_replica.cc:330] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: stopping tablet replica
I20260924 03:08:38.829312 2682 raft_consensus.cc:2229] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:38.829416 2682 raft_consensus.cc:2258] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:38.829849 2635 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: No bootstrap required, opened a new log
I20260924 03:08:38.829900 2635 ts_tablet_manager.cc:1397] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 03:08:38.830029 2682 ts_tablet_manager.cc:1905] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:38.830024 2635 raft_consensus.cc:348] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.830085 2635 raft_consensus.cc:374] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.830107 2635 raft_consensus.cc:729] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188e1648da9e43009f170d38e3099640, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.830147 2635 consensus_queue.cc:260] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.830255 2635 ts_tablet_manager.cc:1428] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:38.831714 2682 ts_tablet_manager.cc:1918] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:38.831806 2637 tablet_bootstrap.cc:492] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: No bootstrap required, opened a new log
I20260924 03:08:38.831854 2637 ts_tablet_manager.cc:1397] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260924 03:08:38.831826 2682 log.cc:1196] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/c3e99bcb475e4291a080b30b78be5061
I20260924 03:08:38.832008 2637 raft_consensus.cc:348] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.832103 2637 raft_consensus.cc:374] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:38.832141 2637 raft_consensus.cc:729] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Initialized, Role: FOLLOWER
I20260924 03:08:38.832182 2637 consensus_queue.cc:260] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:38.832322 2637 ts_tablet_manager.cc:1428] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:38.832441 2682 ts_tablet_manager.cc:1939] T c3e99bcb475e4291a080b30b78be5061 P 8003d51534d748d39c2182de212ebaab: Deleting consensus metadata
I20260924 03:08:38.833093 2113 catalog_manager.cc:4841] TS 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481): tablet c3e99bcb475e4291a080b30b78be5061 (table test-table [id=c934e32b7e7a4e6a914fdb6ecad7de6a]) successfully deleted
I20260924 03:08:39.026161 2642 raft_consensus.cc:482] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:39.026300 2642 raft_consensus.cc:504] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:39.026654 2642 leader_election.cc:288] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:39.029425 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.026859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39481 (local address 127.0.98.3:50909)
0924 03:08:39.026917 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:39.026928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:39.026970 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:39.027141 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:39.027142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:39.027143 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:39.027215 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.027217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.027860 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.027861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.028589 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.028591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.029155 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.029156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.029233 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.029236 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:39.029309 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:08:39.029350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:08:39.029417 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.026942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50909 (local address 127.0.98.2:39481)
0924 03:08:39.027011 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:39.027014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:39.027022 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:39.027046 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:39.027047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:39.027055 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:39.027123 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:39.027291 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.027773 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.028702 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.029063 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.029154 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.029164 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:39.029318 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:08:39.029350 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:08:39.029682 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d362ce7122348c3bb710007040a8b5a" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab" is_pre_election: true
I20260924 03:08:39.029773 2426 raft_consensus.cc:2454] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:39.029942 2497 leader_election.cc:302] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:39.030061 2642 raft_consensus.cc:2781] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:39.030103 2642 raft_consensus.cc:482] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:39.030138 2642 raft_consensus.cc:3037] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:39.030507 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.026920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43917 (local address 127.0.98.1:37311)
0924 03:08:39.026980 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:39.026982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:39.027478 (+ 496us) server_negotiation.cc:407] Connection header received
0924 03:08:39.027610 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:39.027611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:39.027620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:39.027700 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:39.027858 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.028294 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.029641 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.030170 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.030258 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.030267 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:39.030413 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:08:39.030449 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:08:39.030526 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.026859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.98.3:43917)
0924 03:08:39.027034 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:39.027042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:39.027494 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:39.027706 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:39.027706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:39.027708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:39.027784 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.027786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.028377 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.028378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.029492 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.029495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.030257 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.030258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.030329 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.030337 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:08:39.030406 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:08:39.030443 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260924 03:08:39.030717 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d362ce7122348c3bb710007040a8b5a" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640" is_pre_election: true
I20260924 03:08:39.030797 2286 raft_consensus.cc:2454] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:39.030997 2642 raft_consensus.cc:504] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:39.031126 2642 leader_election.cc:288] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Requested vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:39.031240 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d362ce7122348c3bb710007040a8b5a" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640"
I20260924 03:08:39.031318 2286 raft_consensus.cc:3037] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:39.031287 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d362ce7122348c3bb710007040a8b5a" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab"
I20260924 03:08:39.031358 2426 raft_consensus.cc:3037] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:39.031888 2426 raft_consensus.cc:2454] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:39.031942 2286 raft_consensus.cc:2454] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:39.032073 2497 leader_election.cc:302] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:39.032148 2642 raft_consensus.cc:2781] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:39.032264 2642 raft_consensus.cc:686] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Becoming Leader. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Running, Role: LEADER
I20260924 03:08:39.032326 2642 consensus_queue.cc:237] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:39.032949 2129 catalog_manager.cc:5495] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } health_report { overall_health: HEALTHY } } }
I20260924 03:08:39.050290 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.048436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.0.98.3:32843)
0924 03:08:39.048471 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:39.048473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:39.048480 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:39.048534 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:39.048534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:39.048550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:39.048601 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:39.048673 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.049090 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.049697 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.049854 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.049899 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.050010 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:39.050102 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:39.050153 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:39.050169 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 03:08:39.050228 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:39.054272 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.052301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.98.3:51619)
0924 03:08:39.052372 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:39.052383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:39.052417 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:39.052531 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:39.052531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:39.052532 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:39.052579 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.052580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.053101 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.053101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.053703 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.053705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.054008 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.054009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.054053 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.054063 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:39.054166 (+ 103us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:39.054167 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:39.054205 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 03:08:39.054229 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:08:39.054299 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.052388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51619 (local address 127.0.98.1:37311)
0924 03:08:39.052423 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:39.052425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:39.052433 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:39.052461 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:39.052461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:39.052470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:39.052527 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:39.052632 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.053040 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.053787 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.053950 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.053999 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.054116 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:39.054137 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:39.054167 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:39.054233 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:08:39.054256 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:39.056062 2426 raft_consensus.cc:1261] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:39.056107 2286 raft_consensus.cc:1261] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:39.056326 2689 consensus_queue.cc:1035] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:39.056437 2642 consensus_queue.cc:1035] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:39.063282 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.061151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54335 (local address 127.0.98.3:32843)
0924 03:08:39.061186 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:39.061188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:39.061194 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:08:39.061213 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:39.061214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:39.061222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:39.061266 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:39.061401 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.061839 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.062587 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.062841 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.062909 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.063068 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:39.063103 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:39.063171 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:39.063212 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 03:08:39.063226 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:39.063289 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:39.061067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32843 (local address 127.0.98.1:54335)
0924 03:08:39.061105 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:39.061116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:39.061152 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:39.061303 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:39.061304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:39.061305 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:08:39.061336 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.061337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.061907 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.061907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.062517 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:39.062519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:39.062922 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:39.062923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:39.062982 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:39.062988 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:08:39.063175 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:08:39.063176 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:39.063210 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 03:08:39.063246 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 03:08:39.063620 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:08:40.581037 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:40.581138 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:40.581898 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Sending DeleteTablet for 3 replicas of tablet 4d362ce7122348c3bb710007040a8b5a
I20260924 03:08:40.582194 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d362ce7122348c3bb710007040a8b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38346
I20260924 03:08:40.582170 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d362ce7122348c3bb710007040a8b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51982
I20260924 03:08:40.582261 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d362ce7122348c3bb710007040a8b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41302
I20260924 03:08:40.582396 2720 tablet_replica.cc:330] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: stopping tablet replica
I20260924 03:08:40.582473 2720 raft_consensus.cc:2229] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:40.582512 2720 raft_consensus.cc:2258] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.582757 2719 tablet_replica.cc:330] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: stopping tablet replica
I20260924 03:08:40.582818 2719 raft_consensus.cc:2229] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:40.582851 2719 raft_consensus.cc:2258] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.583083 2719 ts_tablet_manager.cc:1905] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.584266 2720 ts_tablet_manager.cc:1905] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.584903 2719 ts_tablet_manager.cc:1918] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:08:40.584969 2719 log.cc:1196] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/4d362ce7122348c3bb710007040a8b5a
I20260924 03:08:40.585189 2719 ts_tablet_manager.cc:1939] T 4d362ce7122348c3bb710007040a8b5a P 188e1648da9e43009f170d38e3099640: Deleting consensus metadata
I20260924 03:08:40.585601 2116 catalog_manager.cc:4841] TS 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311): tablet 4d362ce7122348c3bb710007040a8b5a (table test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]) successfully deleted
I20260924 03:08:40.585801 2720 ts_tablet_manager.cc:1918] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:08:40.585868 2720 log.cc:1196] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/4d362ce7122348c3bb710007040a8b5a
I20260924 03:08:40.586064 2720 ts_tablet_manager.cc:1939] T 4d362ce7122348c3bb710007040a8b5a P 8003d51534d748d39c2182de212ebaab: Deleting consensus metadata
I20260924 03:08:40.586267 2721 tablet_replica.cc:330] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: stopping tablet replica
I20260924 03:08:40.586377 2721 raft_consensus.cc:2229] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:40.586453 2113 catalog_manager.cc:4841] TS 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481): tablet 4d362ce7122348c3bb710007040a8b5a (table test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]) successfully deleted
I20260924 03:08:40.586476 2721 raft_consensus.cc:2258] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.586670 2721 ts_tablet_manager.cc:1905] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.588143 2721 ts_tablet_manager.cc:1918] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:08:40.588191 2721 log.cc:1196] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/4d362ce7122348c3bb710007040a8b5a
I20260924 03:08:40.588421 2721 ts_tablet_manager.cc:1939] T 4d362ce7122348c3bb710007040a8b5a P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting consensus metadata
I20260924 03:08:40.588750 2113 catalog_manager.cc:4841] TS af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843): tablet 4d362ce7122348c3bb710007040a8b5a (table test-table [id=fd9f5b3ca3614b1fb83a2aca8201b451]) successfully deleted
Sep 24 03:08:40 dist-test-slave-jtnz krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219320, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:08:40 dist-test-slave-jtnz krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219320, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:40.604650 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:40.598045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.98.62:46869)
0924 03:08:40.598173 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:40.598174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:40.598218 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:08:40.598281 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:40.598282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:40.598291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:40.598353 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:40.598845 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:40.599429 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:40.600148 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:40.600372 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:40.600423 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:40.602534 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:40.602535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:40.602535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:40.602541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:40.603789 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:40.604086 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:40.604086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:40.604087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:40.604135 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:40.604290 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:40.604291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:40.604291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:40.604381 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:40.604452 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:40.604534 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:08:40.604589 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20260924 03:08:40.605882 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:40.606016 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:40.607261 2266 tablet_service.cc:1467] Processing CreateTablet for tablet d774859b89f44170935dca8827439ad5 (DEFAULT_TABLE table=test-table [id=791328d94b174064afb46ab2c1a95985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.607286 2406 tablet_service.cc:1467] Processing CreateTablet for tablet d774859b89f44170935dca8827439ad5 (DEFAULT_TABLE table=test-table [id=791328d94b174064afb46ab2c1a95985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.607396 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774859b89f44170935dca8827439ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.607399 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774859b89f44170935dca8827439ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.608333 2730 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Bootstrap starting.
I20260924 03:08:40.608369 2731 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Bootstrap starting.
I20260924 03:08:40.608489 2546 tablet_service.cc:1467] Processing CreateTablet for tablet d774859b89f44170935dca8827439ad5 (DEFAULT_TABLE table=test-table [id=791328d94b174064afb46ab2c1a95985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.608616 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774859b89f44170935dca8827439ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.608727 2730 tablet_bootstrap.cc:654] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.608825 2731 tablet_bootstrap.cc:654] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.609289 2730 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: No bootstrap required, opened a new log
I20260924 03:08:40.609349 2730 ts_tablet_manager.cc:1397] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.609407 2731 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: No bootstrap required, opened a new log
I20260924 03:08:40.609462 2731 ts_tablet_manager.cc:1397] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.609498 2730 raft_consensus.cc:348] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.609575 2730 raft_consensus.cc:374] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.609598 2730 raft_consensus.cc:729] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.609576 2731 raft_consensus.cc:348] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.609638 2731 raft_consensus.cc:374] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.609681 2731 raft_consensus.cc:729] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188e1648da9e43009f170d38e3099640, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.609665 2730 consensus_queue.cc:260] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.609730 2731 consensus_queue.cc:260] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.609834 2731 ts_tablet_manager.cc:1428] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:40.609834 2730 ts_tablet_manager.cc:1428] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:40.610380 2735 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Bootstrap starting.
I20260924 03:08:40.610803 2735 tablet_bootstrap.cc:654] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.611382 2735 tablet_bootstrap.cc:492] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: No bootstrap required, opened a new log
I20260924 03:08:40.611451 2735 ts_tablet_manager.cc:1397] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.611588 2735 raft_consensus.cc:348] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.611662 2735 raft_consensus.cc:374] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.611685 2735 raft_consensus.cc:729] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.611740 2735 consensus_queue.cc:260] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.611837 2735 ts_tablet_manager.cc:1428] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 03:08:40.817663 2613 tablet.cc:2369] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:40.879712 2689 raft_consensus.cc:482] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:40.879793 2689 raft_consensus.cc:504] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.879935 2689 leader_election.cc:288] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:40.880115 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774859b89f44170935dca8827439ad5" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640" is_pre_election: true
I20260924 03:08:40.880152 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774859b89f44170935dca8827439ad5" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab" is_pre_election: true
I20260924 03:08:40.880223 2426 raft_consensus.cc:2454] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:40.880239 2286 raft_consensus.cc:2454] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:40.880378 2497 leader_election.cc:302] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:40.880483 2689 raft_consensus.cc:2781] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:40.880549 2689 raft_consensus.cc:482] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:40.880585 2689 raft_consensus.cc:3037] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:40.881232 2689 raft_consensus.cc:504] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.881366 2689 leader_election.cc:288] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Requested vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:40.881508 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774859b89f44170935dca8827439ad5" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab"
I20260924 03:08:40.881513 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774859b89f44170935dca8827439ad5" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640"
I20260924 03:08:40.881587 2426 raft_consensus.cc:3037] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:40.881605 2286 raft_consensus.cc:3037] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:40.882241 2426 raft_consensus.cc:2454] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:40.882277 2286 raft_consensus.cc:2454] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:40.882405 2497 leader_election.cc:302] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:40.882510 2689 raft_consensus.cc:2781] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:40.882576 2689 raft_consensus.cc:686] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Becoming Leader. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Running, Role: LEADER
I20260924 03:08:40.882632 2689 consensus_queue.cc:237] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.883278 2128 catalog_manager.cc:5495] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } health_report { overall_health: HEALTHY } } }
I20260924 03:08:40.894726 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:40.892827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.0.98.3:32843)
0924 03:08:40.892954 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:40.892955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:40.892961 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:08:40.892978 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:40.892979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:40.892988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:40.893046 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:40.893145 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:40.893564 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:40.894216 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:40.894396 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:40.894439 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:40.894546 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:40.894569 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:40.894602 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:40.894635 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 03:08:40.894650 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 03:08:40.896454 2286 raft_consensus.cc:1261] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:40.896481 2426 raft_consensus.cc:1261] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:40.896648 2689 consensus_queue.cc:1035] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:40.896739 2709 consensus_queue.cc:1035] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:40.906330 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:40.906422 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:40.906958 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Sending DeleteTablet for 3 replicas of tablet d774859b89f44170935dca8827439ad5
I20260924 03:08:40.907160 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet d774859b89f44170935dca8827439ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51982
I20260924 03:08:40.907178 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet d774859b89f44170935dca8827439ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38346
I20260924 03:08:40.907183 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet d774859b89f44170935dca8827439ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41302
I20260924 03:08:40.907269 2720 tablet_replica.cc:330] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: stopping tablet replica
I20260924 03:08:40.907269 2719 tablet_replica.cc:330] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: stopping tablet replica
I20260924 03:08:40.907269 2721 tablet_replica.cc:330] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: stopping tablet replica
I20260924 03:08:40.907313 2720 raft_consensus.cc:2229] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:40.907312 2721 raft_consensus.cc:2229] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:40.907315 2719 raft_consensus.cc:2229] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:40.907342 2720 raft_consensus.cc:2258] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.907346 2719 raft_consensus.cc:2258] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.907351 2721 raft_consensus.cc:2258] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:40.907577 2719 ts_tablet_manager.cc:1905] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.907618 2720 ts_tablet_manager.cc:1905] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.907603 2721 ts_tablet_manager.cc:1905] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:40.909067 2719 ts_tablet_manager.cc:1918] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:40.909098 2720 ts_tablet_manager.cc:1918] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:40.909137 2720 log.cc:1196] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/d774859b89f44170935dca8827439ad5
I20260924 03:08:40.909135 2719 log.cc:1196] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/d774859b89f44170935dca8827439ad5
I20260924 03:08:40.909296 2720 ts_tablet_manager.cc:1939] T d774859b89f44170935dca8827439ad5 P 8003d51534d748d39c2182de212ebaab: Deleting consensus metadata
I20260924 03:08:40.909376 2719 ts_tablet_manager.cc:1939] T d774859b89f44170935dca8827439ad5 P 188e1648da9e43009f170d38e3099640: Deleting consensus metadata
I20260924 03:08:40.909438 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:40.909592 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:40.909694 2116 catalog_manager.cc:4841] TS 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311): tablet d774859b89f44170935dca8827439ad5 (table test-table [id=791328d94b174064afb46ab2c1a95985]) successfully deleted
I20260924 03:08:40.909746 2113 catalog_manager.cc:4841] TS 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481): tablet d774859b89f44170935dca8827439ad5 (table test-table [id=791328d94b174064afb46ab2c1a95985]) successfully deleted
I20260924 03:08:40.909791 2721 ts_tablet_manager.cc:1918] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:08:40.909864 2721 log.cc:1196] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/d774859b89f44170935dca8827439ad5
I20260924 03:08:40.910053 2721 ts_tablet_manager.cc:1939] T d774859b89f44170935dca8827439ad5 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting consensus metadata
I20260924 03:08:40.910336 2113 catalog_manager.cc:4841] TS af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843): tablet d774859b89f44170935dca8827439ad5 (table test-table [id=791328d94b174064afb46ab2c1a95985]) successfully deleted
I20260924 03:08:40.910991 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 19c08eaaca29424f98ef97b4d715b179 (DEFAULT_TABLE table=test-table [id=0d48374e48bc4905b170134f1012b179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.911007 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 19c08eaaca29424f98ef97b4d715b179 (DEFAULT_TABLE table=test-table [id=0d48374e48bc4905b170134f1012b179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.911033 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 19c08eaaca29424f98ef97b4d715b179 (DEFAULT_TABLE table=test-table [id=0d48374e48bc4905b170134f1012b179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:40.911094 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c08eaaca29424f98ef97b4d715b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.911109 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c08eaaca29424f98ef97b4d715b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.911128 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c08eaaca29424f98ef97b4d715b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:40.911866 2730 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Bootstrap starting.
I20260924 03:08:40.912209 2730 tablet_bootstrap.cc:654] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.912211 2735 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Bootstrap starting.
I20260924 03:08:40.912497 2735 tablet_bootstrap.cc:654] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.912699 2730 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: No bootstrap required, opened a new log
I20260924 03:08:40.912750 2731 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640: Bootstrap starting.
I20260924 03:08:40.912758 2730 ts_tablet_manager.cc:1397] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.912856 2730 raft_consensus.cc:348] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.912926 2730 raft_consensus.cc:374] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.912945 2730 raft_consensus.cc:729] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8003d51534d748d39c2182de212ebaab, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.912995 2735 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: No bootstrap required, opened a new log
I20260924 03:08:40.913004 2730 consensus_queue.cc:260] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.913033 2735 ts_tablet_manager.cc:1397] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.913089 2731 tablet_bootstrap.cc:654] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:40.913108 2730 ts_tablet_manager.cc:1428] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:40.913143 2735 raft_consensus.cc:348] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.913208 2735 raft_consensus.cc:374] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.913237 2735 raft_consensus.cc:729] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.913270 2735 consensus_queue.cc:260] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.913349 2735 ts_tablet_manager.cc:1428] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:40.913787 2731 tablet_bootstrap.cc:492] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640: No bootstrap required, opened a new log
I20260924 03:08:40.913865 2731 ts_tablet_manager.cc:1397] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:40.913964 2731 raft_consensus.cc:348] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.914068 2731 raft_consensus.cc:374] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:40.914090 2731 raft_consensus.cc:729] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188e1648da9e43009f170d38e3099640, State: Initialized, Role: FOLLOWER
I20260924 03:08:40.914134 2731 consensus_queue.cc:260] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:40.914256 2731 ts_tablet_manager.cc:1428] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:41.016803 2743 raft_consensus.cc:482] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:41.016880 2743 raft_consensus.cc:504] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:41.017019 2743 leader_election.cc:288] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:41.017174 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c08eaaca29424f98ef97b4d715b179" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640" is_pre_election: true
I20260924 03:08:41.017263 2286 raft_consensus.cc:2454] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:41.017204 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c08eaaca29424f98ef97b4d715b179" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab" is_pre_election: true
I20260924 03:08:41.017295 2426 raft_consensus.cc:2454] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 0.
I20260924 03:08:41.017418 2500 leader_election.cc:302] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188e1648da9e43009f170d38e3099640, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:41.017544 2743 raft_consensus.cc:2781] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:41.017598 2743 raft_consensus.cc:482] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:41.017648 2743 raft_consensus.cc:3037] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:41.018330 2743 raft_consensus.cc:504] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:41.018498 2743 leader_election.cc:288] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Requested vote from peers 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311), 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481)
I20260924 03:08:41.018643 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c08eaaca29424f98ef97b4d715b179" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8003d51534d748d39c2182de212ebaab"
I20260924 03:08:41.018646 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c08eaaca29424f98ef97b4d715b179" candidate_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188e1648da9e43009f170d38e3099640"
I20260924 03:08:41.018723 2286 raft_consensus.cc:3037] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:41.018723 2426 raft_consensus.cc:3037] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:41.019443 2286 raft_consensus.cc:2454] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:41.019443 2426 raft_consensus.cc:2454] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9ac651c0fd41b1ad1bfec96bc30e91 in term 1.
I20260924 03:08:41.019644 2497 leader_election.cc:302] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8003d51534d748d39c2182de212ebaab, af9ac651c0fd41b1ad1bfec96bc30e91; no voters:
I20260924 03:08:41.019747 2743 raft_consensus.cc:2781] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:41.019820 2743 raft_consensus.cc:686] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Becoming Leader. State: Replica: af9ac651c0fd41b1ad1bfec96bc30e91, State: Running, Role: LEADER
I20260924 03:08:41.019881 2743 consensus_queue.cc:237] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } }
I20260924 03:08:41.020378 2129 catalog_manager.cc:5495] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9ac651c0fd41b1ad1bfec96bc30e91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32843 } health_report { overall_health: HEALTHY } } }
I20260924 03:08:41.029146 2286 raft_consensus.cc:1261] T 19c08eaaca29424f98ef97b4d715b179 P 188e1648da9e43009f170d38e3099640 [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:41.029151 2426 raft_consensus.cc:1261] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Refusing update from remote peer af9ac651c0fd41b1ad1bfec96bc30e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:41.029333 2743 consensus_queue.cc:1035] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8003d51534d748d39c2182de212ebaab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:41.029397 2689 consensus_queue.cc:1035] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "188e1648da9e43009f170d38e3099640" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:08:41.033066 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:08:42.547735 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:42.547847 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:42.548600 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75b0217fde24213a7b492660768da1f: Sending DeleteTablet for 3 replicas of tablet 19c08eaaca29424f98ef97b4d715b179
I20260924 03:08:42.548857 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c08eaaca29424f98ef97b4d715b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38346
I20260924 03:08:42.548866 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c08eaaca29424f98ef97b4d715b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41302
I20260924 03:08:42.548872 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c08eaaca29424f98ef97b4d715b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51982
I20260924 03:08:42.548962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2201
I20260924 03:08:42.550326 2772 tablet_replica.cc:330] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: stopping tablet replica
I20260924 03:08:42.550416 2772 raft_consensus.cc:2229] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:42.550460 2772 raft_consensus.cc:2258] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:42.550561 2773 tablet_replica.cc:330] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: stopping tablet replica
I20260924 03:08:42.550633 2773 raft_consensus.cc:2229] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:42.550690 2773 raft_consensus.cc:2258] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:42.550901 2773 ts_tablet_manager.cc:1905] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:42.550931 2772 ts_tablet_manager.cc:1905] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:42.552414 2773 ts_tablet_manager.cc:1918] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:08:42.552480 2773 log.cc:1196] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/19c08eaaca29424f98ef97b4d715b179
I20260924 03:08:42.552708 2773 ts_tablet_manager.cc:1939] T 19c08eaaca29424f98ef97b4d715b179 P af9ac651c0fd41b1ad1bfec96bc30e91: Deleting consensus metadata
I20260924 03:08:42.552805 2772 ts_tablet_manager.cc:1918] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:08:42.552863 2772 log.cc:1196] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/19c08eaaca29424f98ef97b4d715b179
I20260924 03:08:42.552986 2113 catalog_manager.cc:4841] TS af9ac651c0fd41b1ad1bfec96bc30e91 (127.0.98.3:32843): tablet 19c08eaaca29424f98ef97b4d715b179 (table test-table [id=0d48374e48bc4905b170134f1012b179]) successfully deleted
I20260924 03:08:42.553115 2772 ts_tablet_manager.cc:1939] T 19c08eaaca29424f98ef97b4d715b179 P 8003d51534d748d39c2182de212ebaab: Deleting consensus metadata
I20260924 03:08:42.553432 2113 catalog_manager.cc:4841] TS 8003d51534d748d39c2182de212ebaab (127.0.98.2:39481): tablet 19c08eaaca29424f98ef97b4d715b179 (table test-table [id=0d48374e48bc4905b170134f1012b179]) successfully deleted
W20260924 03:08:42.555550 2116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37311 (error 108)
I20260924 03:08:42.555855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2341
I20260924 03:08:42.556479 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:42.555880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37311 (local address 127.0.0.1:51992)
0924 03:08:42.556391 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:42.556437 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":86,"threads_started":1}
W20260924 03:08:42.556591 2116 catalog_manager.cc:4584] TS 188e1648da9e43009f170d38e3099640 (127.0.98.1:37311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19c08eaaca29424f98ef97b4d715b179: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37311: connect: Connection refused (error 111)
I20260924 03:08:42.561447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2481
I20260924 03:08:42.566671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2097
2026-09-24T03:08:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5686 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:42 dist-test-slave-jtnz krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:42 dist-test-slave-jtnz krb5kdc[2779](info): set up 2 sockets
Sep 24 03:08:42 dist-test-slave-jtnz krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260924 03:08:43.545768 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:43 dist-test-slave-jtnz krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:43.649150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:43.722030 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:43.722246 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:43.722283 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:43.723364 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:43.723423 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:43.723443 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:43.723464 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:43.723487 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:43.724817 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:43.725045 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:43.725243 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:43.729887 2792 server_base.cc:1034] running on GCE node
W20260924 03:08:43.729921 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:43.729863 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:43.729871 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:43.730289 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:43.730553 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:43.731684 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790219323731673 us; error 26 us; skew 500 ppm
Sep 24 03:08:43 dist-test-slave-jtnz krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:43.733733 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:43.734107 2792 webserver.cc:460] Webserver started at http://127.0.98.62:45363/ using document root <none> and password file <none>
I20260924 03:08:43.734310 2792 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:43.734349 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:43.734467 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:43.735291 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "cc268c6dbf2342d6823a055b4fec3c97"
format_stamp: "Formatted at 2026-09-24 03:08:43 on dist-test-slave-jtnz"
server_key: "52da1e69e8d8e652cd9d27d5d26f618b"
server_key_iv: "bb0e6831bac5e984a31487fd6d6cb957"
server_key_version: "encryptionkey@0"
I20260924 03:08:43.735589 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc268c6dbf2342d6823a055b4fec3c97"
format_stamp: "Formatted at 2026-09-24 03:08:43 on dist-test-slave-jtnz"
server_key: "52da1e69e8d8e652cd9d27d5d26f618b"
server_key_iv: "bb0e6831bac5e984a31487fd6d6cb957"
server_key_version: "encryptionkey@0"
I20260924 03:08:43.736642 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:43.737208 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:43.737334 2792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:43.737408 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "cc268c6dbf2342d6823a055b4fec3c97"
format_stamp: "Formatted at 2026-09-24 03:08:43 on dist-test-slave-jtnz"
server_key: "52da1e69e8d8e652cd9d27d5d26f618b"
server_key_iv: "bb0e6831bac5e984a31487fd6d6cb957"
server_key_version: "encryptionkey@0"
I20260924 03:08:43.737452 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:43.758016 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:43.759644 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:43.759739 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:43.763077 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42759
I20260924 03:08:43.763154 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42759 every 8 connection(s)
I20260924 03:08:43.763540 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:43.764132 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:43.766044 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97: Bootstrap starting.
I20260924 03:08:43.766585 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:43.766851 2860 log.cc:824] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:43.767458 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97: No bootstrap required, opened a new log
I20260924 03:08:43.768836 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260924 03:08:43.768975 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:43.769017 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc268c6dbf2342d6823a055b4fec3c97, State: Initialized, Role: FOLLOWER
I20260924 03:08:43.769187 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260924 03:08:43.769287 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:43.769327 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:43.769375 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:43.770246 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260924 03:08:43.770351 2860 leader_election.cc:302] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc268c6dbf2342d6823a055b4fec3c97; no voters:
I20260924 03:08:43.770507 2860 leader_election.cc:288] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:43.770540 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:43.770690 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [term 1 LEADER]: Becoming Leader. State: Replica: cc268c6dbf2342d6823a055b4fec3c97, State: Running, Role: LEADER
I20260924 03:08:43.770750 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:43.770865 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } }
I20260924 03:08:43.771189 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc268c6dbf2342d6823a055b4fec3c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } } }
I20260924 03:08:43.771193 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc268c6dbf2342d6823a055b4fec3c97. Latest consensus state: current_term: 1 leader_uuid: "cc268c6dbf2342d6823a055b4fec3c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc268c6dbf2342d6823a055b4fec3c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42759 } } }
I20260924 03:08:43.771340 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:43.771347 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:43.771780 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:43.772233 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:43.773200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2792
I20260924 03:08:43.773276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:43.773411 392 external_mini_cluster.cc:1428] Setting key 78f03443c2f2cc78e7b70dfff8454ba1
I20260924 03:08:43.773552 2872 catalog_manager.cc:1347] Generated new cluster ID: 5e98817d8a6c4b839ad0b1a1ceeb9ea0
I20260924 03:08:43.773605 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:43.781956 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:43.782450 2872 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 03:08:43 dist-test-slave-jtnz krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:43.789863 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc268c6dbf2342d6823a055b4fec3c97: Generated new TSK 0
I20260924 03:08:43.790012 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:43.792419 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:43.773752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.98.62:42759)
0924 03:08:43.778610 (+ 4858us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:43.778618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:43.778646 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:08:43.778745 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:43.778752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:43.778821 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:43.778934 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:43.779724 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:43.780681 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:43.781367 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:43.781862 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:43.781985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:43.789094 (+ 7109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:43.789099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:43.789099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:43.789110 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:43.790868 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:43.791251 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:43.791253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:43.791253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:43.791327 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:43.791527 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:43.791528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:43.791528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:43.791706 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:43.791821 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:43.791978 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:08:43.792077 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":4808,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 03:08:43.976773 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.183s user 0.000s sys 0.003s
I20260924 03:08:43.977124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:44.053876 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:44.054085 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:44.054119 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:44.055243 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:44.055335 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:44.055382 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:44.056860 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:44.057112 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:44.057312 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:44.062153 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:44.062170 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:44.062230 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:44.062234 2886 server_base.cc:1034] running on GCE node
I20260924 03:08:44.062533 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:44.062796 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:44.063971 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790219324063958 us; error 25 us; skew 500 ppm
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:44.066133 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:44.066588 2886 webserver.cc:460] Webserver started at http://127.0.98.1:43565/ using document root <none> and password file <none>
I20260924 03:08:44.066794 2886 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:44.066857 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:44.067000 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:44.067919 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca1c88343fe44ca39ea281987d16ac07"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "7770ac133aea63aa8294a3571d6d6eab"
server_key_iv: "b8c70910a19583869ba234461a1b288b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.068255 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca1c88343fe44ca39ea281987d16ac07"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "7770ac133aea63aa8294a3571d6d6eab"
server_key_iv: "b8c70910a19583869ba234461a1b288b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.069476 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:44.070117 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.070298 2886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:44.070393 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "ca1c88343fe44ca39ea281987d16ac07"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "7770ac133aea63aa8294a3571d6d6eab"
server_key_iv: "b8c70910a19583869ba234461a1b288b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.070479 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:44.088652 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:44.090158 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:44.090363 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:44.090564 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:44.090806 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:44.090857 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.090891 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:44.090914 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.096014 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45269
I20260924 03:08:44.096050 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45269 every 8 connection(s)
I20260924 03:08:44.096333 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:44.101527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 2886
I20260924 03:08:44.101599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:44.101752 392 external_mini_cluster.cc:1428] Setting key 5d5a863910c04980a8be897d37474481
I20260924 03:08:44.104532 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.096974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36459 (local address 127.0.98.62:42759)
0924 03:08:44.097012 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:44.097014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:44.097204 (+ 190us) server_negotiation.cc:407] Connection header received
0924 03:08:44.097963 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:44.097963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:44.097974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:44.098020 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:44.098646 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.099112 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.099957 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.100125 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.100169 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.101894 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:44.101896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:44.101897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:44.101902 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:44.103366 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.103720 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.103721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.103721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.103770 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.103982 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.103982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.103983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.104082 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:44.104146 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:44.104370 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 03:08:44.104448 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:44.104661 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.096942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.1:36459)
0924 03:08:44.097098 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:44.097134 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:44.097889 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:44.098080 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:44.098083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:44.098406 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:44.098577 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.098581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.099183 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.099183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.099893 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.099895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.100186 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.100187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.100270 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.100622 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:44.100629 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:44.101815 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:44.103452 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.103458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.103460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.103646 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.103827 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.103827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.103828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.103924 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.104140 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:44.104145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:44.104262 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 03:08:44.104351 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 03:08:44.104472 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 03:08:44.105190 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260924 03:08:44.105283 3015 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:44.105445 3015 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260924 03:08:44.105871 2824 ts_manager.cc:194] Registered new tserver with Master: ca1c88343fe44ca39ea281987d16ac07 (127.0.98.1:45269)
I20260924 03:08:44.106506 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36459
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:44.123229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:44.197614 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:44.197801 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:44.197834 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:44.198945 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:44.199018 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:44.199061 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:44.200405 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:44.200623 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:44.200800 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:44.205349 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:44.205335 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:44.205363 3020 server_base.cc:1034] running on GCE node
W20260924 03:08:44.205335 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:44.205713 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:44.205895 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:44.207034 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790219324207029 us; error 34 us; skew 500 ppm
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:44.209188 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:44.209625 3020 webserver.cc:460] Webserver started at http://127.0.98.2:43235/ using document root <none> and password file <none>
I20260924 03:08:44.209821 3020 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:44.209882 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:44.210019 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:44.210973 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d2d3f392ad347b4848a68f6ab14abab"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "28fcfaed28bf49dedf7e7baab049be58"
server_key_iv: "5010b6bc1f30fc3397159d7f0eaa2eb6"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.211287 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d2d3f392ad347b4848a68f6ab14abab"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "28fcfaed28bf49dedf7e7baab049be58"
server_key_iv: "5010b6bc1f30fc3397159d7f0eaa2eb6"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.212426 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:44.213078 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.213236 3020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:44.213323 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "5d2d3f392ad347b4848a68f6ab14abab"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "28fcfaed28bf49dedf7e7baab049be58"
server_key_iv: "5010b6bc1f30fc3397159d7f0eaa2eb6"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.213402 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:44.231662 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:44.233124 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:44.233263 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:44.233449 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:44.233716 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:44.233764 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.233796 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:44.233820 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.239203 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38127
I20260924 03:08:44.239250 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38127 every 8 connection(s)
I20260924 03:08:44.239607 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:44.247340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3020
I20260924 03:08:44.247442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:44.247629 392 external_mini_cluster.cc:1428] Setting key 02d6d0c7029563f4f55451809a639472
I20260924 03:08:44.247718 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.240324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44733 (local address 127.0.98.62:42759)
0924 03:08:44.240378 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:44.240380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:44.240562 (+ 182us) server_negotiation.cc:407] Connection header received
0924 03:08:44.241297 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:44.241298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:44.241306 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:44.241347 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:44.241814 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.242362 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.243246 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.243437 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.243483 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.245307 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:44.245308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:44.245308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:44.245313 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:44.246409 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.246738 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.246739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.246739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.246791 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.247017 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.247018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.247018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.247111 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:44.247158 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:44.247551 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:08:44.247624 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:44.247917 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.240302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.2:44733)
0924 03:08:44.240452 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:44.240480 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:44.241220 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:44.241397 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:44.241399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:44.241608 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:44.241756 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.241759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.242439 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.242440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.243178 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.243179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.243500 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.243500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.243594 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.244038 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:44.244063 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:44.245225 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:44.246480 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.246484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.246486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.246682 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.246857 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.246857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.246857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.246965 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.247202 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:44.247207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:44.247358 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 03:08:44.247513 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:08:44.247677 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 03:08:44.248329 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260924 03:08:44.248415 3149 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:44.248595 3149 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260924 03:08:44.248916 2824 ts_manager.cc:194] Registered new tserver with Master: 5d2d3f392ad347b4848a68f6ab14abab (127.0.98.2:38127)
I20260924 03:08:44.249274 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44733
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:44.268756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:44.344089 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:44.344244 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:44.344261 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:44.345162 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:44.345209 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:44.345245 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:44.346519 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:44.346673 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:44.346879 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:44.351349 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:44.351326 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:44.351362 3154 server_base.cc:1034] running on GCE node
W20260924 03:08:44.351320 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:44.351699 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:44.351917 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:44.353124 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790219324353110 us; error 29 us; skew 500 ppm
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:44.355545 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:44.355926 3154 webserver.cc:460] Webserver started at http://127.0.98.3:37999/ using document root <none> and password file <none>
I20260924 03:08:44.356154 3154 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:44.356220 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:44.356364 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:44.357280 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "6fc048307dcb4073abbeb537c213fcbf"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "489636396927ee1fd51614c28df5d97a"
server_key_iv: "c4772f60f3f44f1f03f1b0d1b0cd394b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.357623 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6fc048307dcb4073abbeb537c213fcbf"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "489636396927ee1fd51614c28df5d97a"
server_key_iv: "c4772f60f3f44f1f03f1b0d1b0cd394b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.358973 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:44.359702 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.359899 3154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:44.359992 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "6fc048307dcb4073abbeb537c213fcbf"
format_stamp: "Formatted at 2026-09-24 03:08:44 on dist-test-slave-jtnz"
server_key: "489636396927ee1fd51614c28df5d97a"
server_key_iv: "c4772f60f3f44f1f03f1b0d1b0cd394b"
server_key_version: "encryptionkey@0"
I20260924 03:08:44.360077 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:44.380489 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:44.382130 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:44.382313 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:44.382511 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:44.382776 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:44.382827 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.382864 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:44.382891 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:44.388485 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32919
I20260924 03:08:44.388522 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32919 every 8 connection(s)
I20260924 03:08:44.388828 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:44.392483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3154
I20260924 03:08:44.392582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:44.392738 392 external_mini_cluster.cc:1428] Setting key 62bc1c13430dc435ff3c3ee8a7dff350
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:44.397571 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.389596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33549 (local address 127.0.98.62:42759)
0924 03:08:44.389670 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:44.389672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:44.389842 (+ 170us) server_negotiation.cc:407] Connection header received
0924 03:08:44.390802 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:44.390803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:44.390816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:44.390872 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:44.391439 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.391917 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.392978 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.393283 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.393350 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.395183 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:44.395184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:44.395185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:44.395191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:44.396560 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.396880 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.396880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.396881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.396945 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:44.397133 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:44.397134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:44.397134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:44.397223 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:44.397270 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:44.397459 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 03:08:44.397514 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:08:44.397754 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:44.389564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42759 (local address 127.0.98.3:33549)
0924 03:08:44.389725 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:44.389765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:44.390700 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:44.390930 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:44.390935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:44.391173 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:44.391330 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.391333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.392002 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.392003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.392861 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:44.392864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:44.393366 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:44.393367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:44.393508 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:44.393906 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:44.393913 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:44.395106 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:44.396625 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.396629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.396630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.396820 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.396999 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:44.396999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:44.397000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:44.397081 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:44.397274 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:44.397278 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:44.397367 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 03:08:44.397437 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:08:44.397540 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260924 03:08:44.398249 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42759
I20260924 03:08:44.398316 3283 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:44.398437 3283 heartbeater.cc:511] Master 127.0.98.62:42759 requested a full tablet report, sending...
I20260924 03:08:44.398741 2824 ts_manager.cc:194] Registered new tserver with Master: 6fc048307dcb4073abbeb537c213fcbf (127.0.98.3:32919)
I20260924 03:08:44.399060 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33549
I20260924 03:08:44.404089 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:08:44.413695 2883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:44.413009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.62:42759)
0924 03:08:44.413056 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:44.413058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:44.413167 (+ 109us) server_negotiation.cc:407] Connection header received
0924 03:08:44.413207 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:44.413207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:44.413215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:44.413259 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:44.413632 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52196: BlockingRecv error: recv got EOF from 127.0.0.1:52196 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:08:44.414608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2886
I20260924 03:08:44.418583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3020
I20260924 03:08:44.422415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3154
I20260924 03:08:44.426218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 2792
2026-09-24T03:08:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1858 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[3295](info): set up 2 sockets
Sep 24 03:08:44 dist-test-slave-jtnz krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260924 03:08:45.381981 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:45.488022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:45.561589 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:45.561764 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:45.561785 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:45.562960 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:45.562999 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:45.563012 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:45.563025 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:45.563069 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:45.564416 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:45.564648 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:45.564868 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:45.569342 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:45.569429 3308 server_base.cc:1034] running on GCE node
W20260924 03:08:45.569379 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:45.569366 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:45.569792 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:45.570060 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:45.571197 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790219325571182 us; error 30 us; skew 500 ppm
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:45.573462 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:45.573899 3308 webserver.cc:460] Webserver started at http://127.0.98.62:41483/ using document root <none> and password file <none>
I20260924 03:08:45.574106 3308 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:45.574149 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:45.574323 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:45.575271 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "2a0d1aca840f419880f39ec3151019b2"
format_stamp: "Formatted at 2026-09-24 03:08:45 on dist-test-slave-jtnz"
server_key: "41feda6acbe42d18d0dea2b8036d0e6b"
server_key_iv: "6fa4119d8101d1d240270c7813f8b889"
server_key_version: "encryptionkey@0"
I20260924 03:08:45.575601 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a0d1aca840f419880f39ec3151019b2"
format_stamp: "Formatted at 2026-09-24 03:08:45 on dist-test-slave-jtnz"
server_key: "41feda6acbe42d18d0dea2b8036d0e6b"
server_key_iv: "6fa4119d8101d1d240270c7813f8b889"
server_key_version: "encryptionkey@0"
I20260924 03:08:45.576833 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:45.577500 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:45.577677 3308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:45.577761 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "2a0d1aca840f419880f39ec3151019b2"
format_stamp: "Formatted at 2026-09-24 03:08:45 on dist-test-slave-jtnz"
server_key: "41feda6acbe42d18d0dea2b8036d0e6b"
server_key_iv: "6fa4119d8101d1d240270c7813f8b889"
server_key_version: "encryptionkey@0"
I20260924 03:08:45.577828 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:45.598002 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:45.599766 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:45.599927 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:45.603645 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42205
I20260924 03:08:45.603679 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42205 every 8 connection(s)
I20260924 03:08:45.604187 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:45.605037 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:45.607155 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2: Bootstrap starting.
I20260924 03:08:45.607710 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:45.608007 3376 log.cc:824] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:45.608649 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2: No bootstrap required, opened a new log
I20260924 03:08:45.610214 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } }
I20260924 03:08:45.610353 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:45.610373 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0d1aca840f419880f39ec3151019b2, State: Initialized, Role: FOLLOWER
I20260924 03:08:45.610538 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } }
I20260924 03:08:45.610599 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:45.610639 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:45.610687 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:45.611439 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } }
I20260924 03:08:45.611546 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0d1aca840f419880f39ec3151019b2; no voters:
I20260924 03:08:45.611706 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:45.611742 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:45.612112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3308
I20260924 03:08:45.612200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:45.612396 392 external_mini_cluster.cc:1428] Setting key 6bd4f040e1ce0732faf4889229472441
I20260924 03:08:45.611889 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0d1aca840f419880f39ec3151019b2, State: Running, Role: LEADER
I20260924 03:08:45.611954 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:45.612591 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } }
I20260924 03:08:45.613152 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0d1aca840f419880f39ec3151019b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } } }
I20260924 03:08:45.613158 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0d1aca840f419880f39ec3151019b2. Latest consensus state: current_term: 1 leader_uuid: "2a0d1aca840f419880f39ec3151019b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0d1aca840f419880f39ec3151019b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42205 } } }
I20260924 03:08:45.613330 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:45.613513 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:45.613685 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:45.614437 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:45.617048 3389 catalog_manager.cc:1347] Generated new cluster ID: 8f9fafedd4bb455997f07a4e19e8ba38
I20260924 03:08:45.617134 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:45.623027 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:45.612818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.0.98.62:42205)
0924 03:08:45.613079 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:45.613088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:45.613120 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:08:45.613241 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:45.613247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:45.613321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:45.613460 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:45.614294 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:45.615561 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:45.616644 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:45.617035 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:45.617155 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:45.619439 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:45.619443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:45.619443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:45.619456 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:45.621531 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:45.621934 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:45.621937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:45.621937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:45.622020 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:45.622259 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:45.622260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:45.622260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:45.622475 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:45.622649 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:45.622748 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 03:08:45.622874 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260924 03:08:45.623853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:08:45.629954 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:45.630699 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:45.633772 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0d1aca840f419880f39ec3151019b2: Generated new TSK 0
I20260924 03:08:45.633945 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:45.639637 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:45.636204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.98.62:42205)
0924 03:08:45.636271 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:45.636273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:45.636398 (+ 125us) server_negotiation.cc:407] Connection header received
0924 03:08:45.636471 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:45.636472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:45.636481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:45.636531 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:45.636865 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:45.637495 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:45.638298 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:45.638505 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:45.638593 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:45.639220 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:45.639221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:45.639221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:45.639384 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:45.639398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:45.639403 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:45.639462 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:45.639530 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:08:45.639557 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:08:45.641824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3308
2026-09-24T03:08:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1213 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3409](info): set up 2 sockets
Sep 24 03:08:45 dist-test-slave-jtnz krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260924 03:08:46.609799 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:46 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:46.711014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:46.782580 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:46.782754 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:46.782774 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:46.783816 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:46.783854 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:46.783867 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:46.783878 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:46.783927 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:46.785141 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:46.785344 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:46.785526 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:46.789672 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:46.789778 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:46.789678 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:46.789793 3422 server_base.cc:1034] running on GCE node
I20260924 03:08:46.790098 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:46.790391 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:46.791517 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790219326791506 us; error 26 us; skew 500 ppm
Sep 24 03:08:46 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:46.793531 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:46.793910 3422 webserver.cc:460] Webserver started at http://127.0.98.62:45945/ using document root <none> and password file <none>
I20260924 03:08:46.794106 3422 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:46.794162 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:46.794409 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:46.795331 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "310322d3a1b94836b1c2dff9576089e9"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "1fb2dda16d81bc04bd6990e8afbed6bd"
server_key_iv: "c578beba9459f188897a6c707ba50696"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.795643 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "310322d3a1b94836b1c2dff9576089e9"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "1fb2dda16d81bc04bd6990e8afbed6bd"
server_key_iv: "c578beba9459f188897a6c707ba50696"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.796825 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:46.797399 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:46.797571 3422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:46.797670 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "310322d3a1b94836b1c2dff9576089e9"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "1fb2dda16d81bc04bd6990e8afbed6bd"
server_key_iv: "c578beba9459f188897a6c707ba50696"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.797734 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:46.812448 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:46.813895 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:46.814036 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:46.817441 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43603
I20260924 03:08:46.817474 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43603 every 8 connection(s)
I20260924 03:08:46.817756 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:46.818634 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:46.820504 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: Bootstrap starting.
I20260924 03:08:46.821000 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:46.821249 3490 log.cc:824] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:46.821833 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: No bootstrap required, opened a new log
I20260924 03:08:46.823348 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260924 03:08:46.823463 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:46.823508 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310322d3a1b94836b1c2dff9576089e9, State: Initialized, Role: FOLLOWER
I20260924 03:08:46.823655 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260924 03:08:46.823714 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:46.823732 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:46.823796 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:46.824542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3422
I20260924 03:08:46.824564 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260924 03:08:46.824641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:46.824656 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310322d3a1b94836b1c2dff9576089e9; no voters:
I20260924 03:08:46.824811 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:46.824828 392 external_mini_cluster.cc:1428] Setting key 3598f78b47ab962e9743bac28594fc97
I20260924 03:08:46.824868 3493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:46.825021 3493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [term 1 LEADER]: Becoming Leader. State: Replica: 310322d3a1b94836b1c2dff9576089e9, State: Running, Role: LEADER
I20260924 03:08:46.825053 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:46.825105 3493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } }
I20260924 03:08:46.825340 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310322d3a1b94836b1c2dff9576089e9. Latest consensus state: current_term: 1 leader_uuid: "310322d3a1b94836b1c2dff9576089e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } } }
I20260924 03:08:46.825418 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:46.825349 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310322d3a1b94836b1c2dff9576089e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310322d3a1b94836b1c2dff9576089e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43603 } } }
I20260924 03:08:46.825503 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9 [sys.catalog]: This master's current role is: LEADER
W20260924 03:08:46.826723 3510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:46.826774 3510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:46.826818 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:46.826988 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:46.828245 3501 catalog_manager.cc:1347] Generated new cluster ID: 35ea72c7f1ad4c2bb590132900dc75d1
I20260924 03:08:46.828293 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:46 dist-test-slave-jtnz krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:46.834985 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:46.825366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.0.98.62:43603)
0924 03:08:46.826250 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:46.826257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:46.826294 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:08:46.826392 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:46.826399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:46.826449 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:46.826585 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:46.827265 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:46.828434 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:46.829145 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:46.829544 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:46.829657 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:46.831826 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:46.831830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:46.831831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:46.831840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:46.833632 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:46.833985 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:46.833987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:46.833987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:46.834080 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:46.834274 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:46.834274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:46.834275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:46.834483 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:46.834625 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:46.834714 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 03:08:46.834847 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":59,"threads_started":1}
I20260924 03:08:46.839004 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:46.839438 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:46.842658 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310322d3a1b94836b1c2dff9576089e9: Generated new TSK 0
I20260924 03:08:46.842793 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:46.869024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:46.977717 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:46.977957 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:46.977994 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:46.979595 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:46.979647 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:46.979682 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:46.981582 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:46.981792 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:46.982041 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:46.986891 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:46.987095 3516 server_base.cc:1034] running on GCE node
W20260924 03:08:46.987396 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:46.987723 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:46.987927 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:46.988143 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:46.989326 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790219326989311 us; error 26 us; skew 500 ppm
Sep 24 03:08:46 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:46.991707 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:46.992154 3516 webserver.cc:460] Webserver started at http://127.0.98.1:41793/ using document root <none> and password file <none>
I20260924 03:08:46.992373 3516 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:46.992436 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:46.992580 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:46.993599 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "1fee6f7f4bd84e459add9e23d7f0b822"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "fcfe3862ae00ba39aef3af823e993ffd"
server_key_iv: "f40c076d2b492dab43fa5997bd3ab485"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.993907 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1fee6f7f4bd84e459add9e23d7f0b822"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "fcfe3862ae00ba39aef3af823e993ffd"
server_key_iv: "f40c076d2b492dab43fa5997bd3ab485"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.995211 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:46.995868 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:46.996042 3516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:46.996126 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "1fee6f7f4bd84e459add9e23d7f0b822"
format_stamp: "Formatted at 2026-09-24 03:08:46 on dist-test-slave-jtnz"
server_key: "fcfe3862ae00ba39aef3af823e993ffd"
server_key_iv: "f40c076d2b492dab43fa5997bd3ab485"
server_key_version: "encryptionkey@0"
I20260924 03:08:46.996197 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:47.008044 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:47.010136 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:47.010310 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:47.010507 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:47.010754 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:47.010800 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.010833 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:47.010856 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.016132 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37279
I20260924 03:08:47.016197 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37279 every 8 connection(s)
I20260924 03:08:47.016471 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:47.023957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3516
I20260924 03:08:47.024072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:47.024288 392 external_mini_cluster.cc:1428] Setting key d6d41248842a901384d985a814b315d7
I20260924 03:08:47.025023 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.017090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46061 (local address 127.0.98.62:43603)
0924 03:08:47.017135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:47.017137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:47.017344 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:08:47.017821 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:47.017822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:47.017836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:47.017911 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:47.018907 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.019401 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.020331 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.020519 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.020587 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.022466 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:47.022467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:47.022467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:47.022473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:47.023603 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.024059 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.024060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.024060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.024140 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.024379 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.024380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.024380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.024536 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:47.024591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:47.024852 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:08:47.024936 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:08:47.025203 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.017072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.1:46061)
0924 03:08:47.017233 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:47.017263 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:47.017723 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:47.017974 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:47.017978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:47.018626 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:47.018803 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.018807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.019491 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.019493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.020238 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.020240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.020591 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.020592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.020704 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.021080 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:47.021087 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:47.022385 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:47.023670 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.023675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.023677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.023964 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.024194 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.024195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.024195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.024309 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.024593 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:47.024600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:47.024727 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 03:08:47.024830 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:08:47.024969 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260924 03:08:47.025792 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260924 03:08:47.025885 3645 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:47.026062 3645 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260924 03:08:47.026595 3454 ts_manager.cc:194] Registered new tserver with Master: 1fee6f7f4bd84e459add9e23d7f0b822 (127.0.98.1:37279)
I20260924 03:08:47.027130 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46061
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:47.047029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:47.121256 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:47.121423 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:47.121443 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:47.122520 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:47.122557 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:47.122583 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:47.123804 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:47.123956 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:47.124102 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:47.128638 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:47.128649 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:47.128649 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:47.128805 3650 server_base.cc:1034] running on GCE node
I20260924 03:08:47.129062 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:47.129281 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:47.130448 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790219327130438 us; error 40 us; skew 500 ppm
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:47.132649 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:47.133111 3650 webserver.cc:460] Webserver started at http://127.0.98.2:44505/ using document root <none> and password file <none>
I20260924 03:08:47.133319 3650 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:47.133381 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:47.133527 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:47.134491 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "33191e566a074437bfab14104a211f4b"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "f50af6e7ee0e31b73988751834f0143f"
server_key_iv: "e8d955d5d19c636066a24cd895509b34"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.134835 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33191e566a074437bfab14104a211f4b"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "f50af6e7ee0e31b73988751834f0143f"
server_key_iv: "e8d955d5d19c636066a24cd895509b34"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.135944 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:47.136534 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.136698 3650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:47.136782 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "33191e566a074437bfab14104a211f4b"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "f50af6e7ee0e31b73988751834f0143f"
server_key_iv: "e8d955d5d19c636066a24cd895509b34"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.136853 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:47.149973 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:47.151481 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:47.151623 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:47.151798 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:47.152045 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:47.152097 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.152132 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:47.152155 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.157346 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34895
I20260924 03:08:47.157388 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34895 every 8 connection(s)
I20260924 03:08:47.157648 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:47.160879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3650
I20260924 03:08:47.160972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:47.161140 392 external_mini_cluster.cc:1428] Setting key df20dccdc4241b9d13a25f321eda3e15
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:47.166296 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.158214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42539 (local address 127.0.98.62:43603)
0924 03:08:47.158257 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:47.158259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:47.158442 (+ 183us) server_negotiation.cc:407] Connection header received
0924 03:08:47.158843 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:47.158844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:47.158854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:47.158901 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:47.159671 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.160112 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.160942 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.161190 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.161270 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.163437 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:47.163438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:47.163438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:47.163444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:47.164765 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.165220 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.165221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.165221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.165267 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.165511 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.165511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.165512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.165596 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:47.165640 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:47.166004 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 03:08:47.166139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:47.166386 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.158181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.2:42539)
0924 03:08:47.158331 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:47.158358 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:47.158768 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:47.158945 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:47.158948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:47.159427 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:47.159594 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.159598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.160180 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.160180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.160853 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.160854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.161279 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.161280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.161410 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.161867 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:47.161874 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:47.163360 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:47.164844 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.164851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.164853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.165137 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.165328 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.165329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.165329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.165451 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.165656 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:47.165662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:47.165812 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 03:08:47.165955 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 03:08:47.166106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260924 03:08:47.166998 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260924 03:08:47.167088 3779 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:47.167258 3779 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260924 03:08:47.167590 3454 ts_manager.cc:194] Registered new tserver with Master: 33191e566a074437bfab14104a211f4b (127.0.98.2:34895)
I20260924 03:08:47.167943 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42539
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:47.183130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:47.253389 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:47.253538 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:47.253557 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:47.254660 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:47.254700 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:47.254729 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:47.256088 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:47.256232 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:47.256439 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:47.260934 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:47.260980 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:47.260974 3784 server_base.cc:1034] running on GCE node
W20260924 03:08:47.260943 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:47.261332 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:47.261543 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:47.262691 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790219327262680 us; error 23 us; skew 500 ppm
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:47.264967 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:47.265373 3784 webserver.cc:460] Webserver started at http://127.0.98.3:36465/ using document root <none> and password file <none>
I20260924 03:08:47.265582 3784 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:47.265642 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:47.265758 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:47.266671 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "80511a3ec14740e582f71b13a3a72df8"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "a5a22e166690dcb849c835af74b03c0b"
server_key_iv: "d9cd011b1094b98afe16a682ff3c2d41"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.267007 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80511a3ec14740e582f71b13a3a72df8"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "a5a22e166690dcb849c835af74b03c0b"
server_key_iv: "d9cd011b1094b98afe16a682ff3c2d41"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.268128 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:47.268780 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.268940 3784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:47.269029 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "80511a3ec14740e582f71b13a3a72df8"
format_stamp: "Formatted at 2026-09-24 03:08:47 on dist-test-slave-jtnz"
server_key: "a5a22e166690dcb849c835af74b03c0b"
server_key_iv: "d9cd011b1094b98afe16a682ff3c2d41"
server_key_version: "encryptionkey@0"
I20260924 03:08:47.269109 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:47.280414 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:47.281946 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:47.282099 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:47.282328 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:47.282603 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:47.282652 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.282685 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:47.282712 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:47.288172 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40949
I20260924 03:08:47.288215 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40949 every 8 connection(s)
I20260924 03:08:47.288491 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:47.296080 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.289141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47939 (local address 127.0.98.62:43603)
0924 03:08:47.289186 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:47.289189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:47.289379 (+ 190us) server_negotiation.cc:407] Connection header received
0924 03:08:47.289756 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:47.289757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:47.289766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:47.289811 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:47.290632 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.291070 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.291892 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.292062 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.292106 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.293679 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:47.293681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:47.293681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:47.293686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:47.295101 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.295411 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.295412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.295412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.295456 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.295633 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.295634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.295634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.295717 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:47.295762 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:47.295943 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:08:47.296003 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:47.296267 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.289137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43603 (local address 127.0.98.3:47939)
0924 03:08:47.289268 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:47.289297 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:47.289685 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:47.289853 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:47.289856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:47.290405 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:47.290560 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.290565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.291139 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.291139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.291828 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:47.291829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.292123 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.292123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:47.292214 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.292541 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:47.292547 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:47.293608 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:47.295164 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.295169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.295171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.295355 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.295506 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:47.295506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:47.295506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:47.295586 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:47.295764 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:47.295767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:47.295855 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 03:08:47.295929 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:08:47.296055 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260924 03:08:47.296680 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43603
I20260924 03:08:47.296756 3913 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:47.296885 3913 heartbeater.cc:511] Master 127.0.98.62:43603 requested a full tablet report, sending...
I20260924 03:08:47.297036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3784
I20260924 03:08:47.297140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:47.297259 3454 ts_manager.cc:194] Registered new tserver with Master: 80511a3ec14740e582f71b13a3a72df8 (127.0.98.3:40949)
I20260924 03:08:47.297324 392 external_mini_cluster.cc:1428] Setting key 8f88043c4cbaf69263e21f855e9a1621
I20260924 03:08:47.297701 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47939
I20260924 03:08:47.298094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219327, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:47.317939 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:47.312293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.0.98.62:43603)
0924 03:08:47.312336 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:47.312338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:47.312425 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:08:47.312463 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:47.312463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:47.312471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:47.312520 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:47.312971 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.313376 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.313997 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:47.314148 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:47.314267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:47.316152 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:47.316153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:47.316153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:47.316158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:47.317174 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.317431 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.317432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.317432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.317472 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:47.317609 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:47.317610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:47.317610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:47.317706 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:47.317768 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:47.317841 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:08:47.317890 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 03:08:47.318123 3453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48480
I20260924 03:08:47.318724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3516
I20260924 03:08:47.323441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3650
I20260924 03:08:47.327653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3784
I20260924 03:08:47.332355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3422
2026-09-24T03:08:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1692 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3925](info): set up 2 sockets
Sep 24 03:08:47 dist-test-slave-jtnz krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260924 03:08:48.281863 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:48.386379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:48.458981 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:48.459203 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:48.459241 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:48.460392 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:48.460457 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:48.460484 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:48.460515 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:48.460538 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:48.461887 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:48.462154 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:48.462366 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:48.466796 3938 server_base.cc:1034] running on GCE node
W20260924 03:08:48.466750 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.466735 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.466768 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.467269 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:48.467571 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:48.468698 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790219328468689 us; error 26 us; skew 500 ppm
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:48.479496 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:48.479920 3938 webserver.cc:460] Webserver started at http://127.0.98.62:46507/ using document root <none> and password file <none>
I20260924 03:08:48.480091 3938 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:48.480129 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:48.480209 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:48.481115 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "e69aaa44a1664e8f8ab27128d5495025"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "1784040816a7cbd66b0fd14dce763298"
server_key_iv: "2a01ee4686a8413dbfa2424bd224c088"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.481395 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "e69aaa44a1664e8f8ab27128d5495025"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "1784040816a7cbd66b0fd14dce763298"
server_key_iv: "2a01ee4686a8413dbfa2424bd224c088"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.482692 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:48.483325 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.483481 3938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.483570 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "e69aaa44a1664e8f8ab27128d5495025"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "1784040816a7cbd66b0fd14dce763298"
server_key_iv: "2a01ee4686a8413dbfa2424bd224c088"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.483637 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:48.495579 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:48.497499 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:48.497638 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:48.501224 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38055
I20260924 03:08:48.501250 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38055 every 8 connection(s)
I20260924 03:08:48.501585 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:48.502341 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:48.504326 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025: Bootstrap starting.
I20260924 03:08:48.504829 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:48.505095 4006 log.cc:824] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:48.505694 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025: No bootstrap required, opened a new log
I20260924 03:08:48.507076 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } }
I20260924 03:08:48.507208 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:48.507249 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69aaa44a1664e8f8ab27128d5495025, State: Initialized, Role: FOLLOWER
I20260924 03:08:48.507386 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } }
I20260924 03:08:48.507470 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:48.507514 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:48.507555 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:48.507972 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } }
I20260924 03:08:48.508070 4006 leader_election.cc:302] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69aaa44a1664e8f8ab27128d5495025; no voters:
I20260924 03:08:48.508235 4006 leader_election.cc:288] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:48.508255 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:48.508442 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:48.508531 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [term 1 LEADER]: Becoming Leader. State: Replica: e69aaa44a1664e8f8ab27128d5495025, State: Running, Role: LEADER
I20260924 03:08:48.508673 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } }
I20260924 03:08:48.509061 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69aaa44a1664e8f8ab27128d5495025. Latest consensus state: current_term: 1 leader_uuid: "e69aaa44a1664e8f8ab27128d5495025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } } }
I20260924 03:08:48.509143 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:48.509068 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69aaa44a1664e8f8ab27128d5495025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aaa44a1664e8f8ab27128d5495025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38055 } } }
I20260924 03:08:48.509203 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:48.509367 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:48.509853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 3938
I20260924 03:08:48.509945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:48.510107 392 external_mini_cluster.cc:1428] Setting key 3dae2e223c8de1fc4125fb67e45c18b2
I20260924 03:08:48.510412 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:48.512004 4018 catalog_manager.cc:1347] Generated new cluster ID: 27313a6ffd0e43e296b4e0b1fcd2ac1f
I20260924 03:08:48.512054 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:48.520151 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:48.520748 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:48.521114 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.510507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.98.62:38055)
0924 03:08:48.510673 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.510681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.510897 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:08:48.511083 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.511090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.511139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.511270 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.511975 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.513348 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.514372 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.514944 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.515029 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.517716 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.517719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.517719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.517729 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.519544 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.519945 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.519947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.519947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.520048 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.520266 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.520267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.520267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.520517 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.520670 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.520757 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.520932 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 03:08:48.522970 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69aaa44a1664e8f8ab27128d5495025: Generated new TSK 0
I20260924 03:08:48.523119 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:48.552181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:48.626138 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:48.626366 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:48.626399 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:48.627413 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:48.627475 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:48.627516 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:48.628916 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:48.629165 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:48.629385 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:48.633836 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.633880 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.633953 4032 server_base.cc:1034] running on GCE node
W20260924 03:08:48.634022 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.634289 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:48.634521 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:48.635677 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790219328635675 us; error 32 us; skew 500 ppm
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:48.637728 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:48.638152 4032 webserver.cc:460] Webserver started at http://127.0.98.1:39005/ using document root <none> and password file <none>
I20260924 03:08:48.638372 4032 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:48.638439 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:48.638602 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:48.639581 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a5a85631d2243939dfbaaf31a8db67f"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "e83a5f226f880c29c06a2aa67d8f2c22"
server_key_iv: "ba1b42b65a40abde8b0e8de43cb4ccea"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.639946 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a5a85631d2243939dfbaaf31a8db67f"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "e83a5f226f880c29c06a2aa67d8f2c22"
server_key_iv: "ba1b42b65a40abde8b0e8de43cb4ccea"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.641085 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:48.641701 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.641988 4032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:08:48.642129 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "9a5a85631d2243939dfbaaf31a8db67f"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "e83a5f226f880c29c06a2aa67d8f2c22"
server_key_iv: "ba1b42b65a40abde8b0e8de43cb4ccea"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.642236 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:48.655293 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:48.656925 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:48.657071 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:48.657259 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:48.657514 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:48.657563 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.657593 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:48.657616 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.663012 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36165
I20260924 03:08:48.663048 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36165 every 8 connection(s)
I20260924 03:08:48.663338 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:48.666392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4032
I20260924 03:08:48.666466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:48.666662 392 external_mini_cluster.cc:1428] Setting key c210750845a22603ea40008c57a50608
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:48.672829 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.663933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38055 (local address 127.0.98.1:42921)
0924 03:08:48.664129 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:48.664171 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:48.664982 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:48.665230 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:48.665233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:48.665458 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:48.665609 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.665613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.666272 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.666273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.667376 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.667379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.667771 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.667772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.667893 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.668354 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:48.668363 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:48.669823 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:48.671632 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.671637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.671639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.671830 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.672035 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.672036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.672036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.672118 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.672393 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:48.672396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:48.672501 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 03:08:48.672594 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:08:48.672693 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260924 03:08:48.672827 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.663994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42921 (local address 127.0.98.62:38055)
0924 03:08:48.664081 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.664083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.664248 (+ 165us) server_negotiation.cc:407] Connection header received
0924 03:08:48.665093 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.665094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.665105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.665168 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.665698 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.666146 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.667467 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.667679 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.667761 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.669904 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.669905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.669906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.669914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.671543 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.671901 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.671903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.671903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.671964 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.672182 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.672183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.672183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.672326 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.672402 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.672642 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.672722 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:08:48.673466 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38055
I20260924 03:08:48.673568 4161 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:48.673728 4161 heartbeater.cc:511] Master 127.0.98.62:38055 requested a full tablet report, sending...
I20260924 03:08:48.674229 3970 ts_manager.cc:194] Registered new tserver with Master: 9a5a85631d2243939dfbaaf31a8db67f (127.0.98.1:36165)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:48.674770 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:48.689999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:48.762990 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:48.763162 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:48.763185 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:48.764281 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:48.764320 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:48.764357 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:48.765725 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:48.765933 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:48.766120 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:48.770680 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.770678 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.770677 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.770692 4166 server_base.cc:1034] running on GCE node
I20260924 03:08:48.771019 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:48.771198 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:48.772326 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790219328772314 us; error 26 us; skew 500 ppm
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:48.774400 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:48.774739 4166 webserver.cc:460] Webserver started at http://127.0.98.2:34633/ using document root <none> and password file <none>
I20260924 03:08:48.774892 4166 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:48.774945 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:48.775027 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:48.775830 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5d5730115c34ed8b60c2822da0429fa"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "83f5a7bcfce51bb6ef62308091547e9e"
server_key_iv: "3c2074877a290a55a572a4a198bc1efa"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.776101 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5d5730115c34ed8b60c2822da0429fa"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "83f5a7bcfce51bb6ef62308091547e9e"
server_key_iv: "3c2074877a290a55a572a4a198bc1efa"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.777235 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:48.777890 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.778057 4166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:48.778132 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "d5d5730115c34ed8b60c2822da0429fa"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "83f5a7bcfce51bb6ef62308091547e9e"
server_key_iv: "3c2074877a290a55a572a4a198bc1efa"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.778219 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:48.790066 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:48.791567 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:48.791657 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:48.791800 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:48.791992 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:48.792018 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.792042 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:48.792054 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.797173 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33581
I20260924 03:08:48.797219 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33581 every 8 connection(s)
I20260924 03:08:48.797456 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:48.803814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4166
I20260924 03:08:48.803937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:48.804117 392 external_mini_cluster.cc:1428] Setting key a9df8d96d6cf319cc5481aaabb7e54b4
I20260924 03:08:48.805364 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.798017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38643 (local address 127.0.98.62:38055)
0924 03:08:48.798073 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.798076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.798253 (+ 177us) server_negotiation.cc:407] Connection header received
0924 03:08:48.799071 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.799072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.799081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.799126 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.799620 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.800086 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.800912 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.801070 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.801113 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.802891 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.802892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.802893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.802898 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.804128 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.804502 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.804503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.804503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.804560 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.804791 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.804792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.804793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.804894 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.804961 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.805206 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.805278 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:08:48.805500 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.797989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38055 (local address 127.0.98.2:38643)
0924 03:08:48.798137 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:48.798163 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:48.798986 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:48.799176 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:48.799179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:48.799387 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:48.799556 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.799560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.800153 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.800154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.800847 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.800849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.801130 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.801131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.801214 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.801540 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:48.801546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:48.802780 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:48.804231 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.804236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.804238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.804429 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.804634 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.804635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.804636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.804725 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.804953 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:48.804959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:48.805084 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 03:08:48.805175 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:08:48.805315 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":98,"threads_started":1}
I20260924 03:08:48.805917 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38055
I20260924 03:08:48.805991 4295 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:48.806170 4295 heartbeater.cc:511] Master 127.0.98.62:38055 requested a full tablet report, sending...
I20260924 03:08:48.806533 3970 ts_manager.cc:194] Registered new tserver with Master: d5d5730115c34ed8b60c2822da0429fa (127.0.98.2:33581)
I20260924 03:08:48.806866 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38643
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:48.824674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:48.897434 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:48.897612 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:48.897643 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:48.898725 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:48.898787 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:48.898826 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:48.900193 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:48.900422 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:48.900631 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:48.905285 4300 server_base.cc:1034] running on GCE node
W20260924 03:08:48.905264 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.905241 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:48.905251 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:48.905597 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:48.905782 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:48.906899 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790219328906880 us; error 34 us; skew 500 ppm
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:48.909250 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:48.909649 4300 webserver.cc:460] Webserver started at http://127.0.98.3:43047/ using document root <none> and password file <none>
I20260924 03:08:48.909870 4300 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:48.909929 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:48.910063 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:48.911034 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "a785a6a9d2b14f8cba27c338739159ae"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "8b77d68832b5fd13da52f9e564772ef9"
server_key_iv: "8b5cd546d432be2b6dcade6666fdff8e"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.911375 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a785a6a9d2b14f8cba27c338739159ae"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "8b77d68832b5fd13da52f9e564772ef9"
server_key_iv: "8b5cd546d432be2b6dcade6666fdff8e"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.912556 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:48.913173 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.913340 4300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:48.913430 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "a785a6a9d2b14f8cba27c338739159ae"
format_stamp: "Formatted at 2026-09-24 03:08:48 on dist-test-slave-jtnz"
server_key: "8b77d68832b5fd13da52f9e564772ef9"
server_key_iv: "8b5cd546d432be2b6dcade6666fdff8e"
server_key_version: "encryptionkey@0"
I20260924 03:08:48.913528 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:48.927452 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:48.929150 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:48.929250 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:48.929397 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:48.929611 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:48.929639 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.929659 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:48.929672 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:48.934842 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36681
I20260924 03:08:48.934873 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36681 every 8 connection(s)
I20260924 03:08:48.935150 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:48.938433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4300
I20260924 03:08:48.938534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:48.938710 392 external_mini_cluster.cc:1428] Setting key a15dfca2189fd739f078d3cf4e5d04d3
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:48.943933 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.935890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59619 (local address 127.0.98.62:38055)
0924 03:08:48.935928 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.935930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.936127 (+ 197us) server_negotiation.cc:407] Connection header received
0924 03:08:48.936941 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.936943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.936954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.937015 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.937545 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.938031 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.939330 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.939567 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.939631 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.941528 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.941529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.941529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.941544 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.942710 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.943088 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.943089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.943090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.943143 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.943375 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.943376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.943376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.943477 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.943554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.943782 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.943850 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:48.944093 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.935867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38055 (local address 127.0.98.3:59619)
0924 03:08:48.936024 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:48.936058 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:48.936847 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:48.937078 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:48.937081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:48.937295 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:48.937444 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.937447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.938120 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.938121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.939257 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:48.939260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.939638 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.939639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:48.939763 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.940230 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:48.940239 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:48.941443 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:48.942788 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.942794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.942796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.942994 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.943216 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:48.943216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:48.943217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:48.943313 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:48.943539 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:48.943545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:48.943667 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 03:08:48.943762 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:08:48.943892 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260924 03:08:48.944455 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38055
I20260924 03:08:48.944515 4429 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:48.944689 4429 heartbeater.cc:511] Master 127.0.98.62:38055 requested a full tablet report, sending...
I20260924 03:08:48.944965 3970 ts_manager.cc:194] Registered new tserver with Master: a785a6a9d2b14f8cba27c338739159ae (127.0.98.3:36681)
I20260924 03:08:48.945303 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59619
I20260924 03:08:48.950306 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:48.969980 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.963345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.0.98.1:36165)
0924 03:08:48.963502 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.963507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.963521 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:08:48.963544 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.963545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.963557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.963618 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.964113 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.964749 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.965402 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.965680 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.965730 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.967625 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.967632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.967632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.967638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.969102 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.969368 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.969370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.969370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.969417 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.969548 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.969549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.969549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.969648 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.969733 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.969793 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.969858 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
Sep 24 03:08:48 dist-test-slave-jtnz krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:48.976923 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:48.970986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.0.98.62:38055)
0924 03:08:48.971052 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:48.971053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:48.971061 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:48.971116 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:48.971117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:48.971125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:48.971173 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:48.971590 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.972046 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.972680 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:48.972860 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:48.972937 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:48.974830 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:48.974831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:48.974831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:48.974838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:48.975908 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.976197 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.976198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.976198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.976245 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:48.976429 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:48.976430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:48.976430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:48.976562 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:48.976651 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:48.976751 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:08:48.976813 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 03:08:48.977149 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40506
I20260924 03:08:48.977458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4032
I20260924 03:08:48.982753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4166
I20260924 03:08:48.986555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4300
I20260924 03:08:48.990271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 3938
2026-09-24T03:08:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1657 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:49 dist-test-slave-jtnz krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:49 dist-test-slave-jtnz krb5kdc[4441](info): set up 2 sockets
Sep 24 03:08:49 dist-test-slave-jtnz krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260924 03:08:49.973616 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:50.081200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.164914 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.165110 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.165146 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.166241 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:50.166298 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.166320 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.166345 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:50.166373 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:50.167686 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:50.167917 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.168145 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.172572 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.172605 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.172595 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.172667 4454 server_base.cc:1034] running on GCE node
I20260924 03:08:50.172950 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.173182 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.174307 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790219330174296 us; error 25 us; skew 500 ppm
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.176512 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:50.176831 4454 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.176873 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.176950 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.177804 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "33ffa0cb11df46549d88c06e431d0659"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "0e800c180984053eacd330d0c12371ef"
server_key_iv: "d7fd316519a320f8881d37d69bfa2d11"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.178074 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "33ffa0cb11df46549d88c06e431d0659"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "0e800c180984053eacd330d0c12371ef"
server_key_iv: "d7fd316519a320f8881d37d69bfa2d11"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.179417 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:08:50.180051 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.180236 4454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:50.180322 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "33ffa0cb11df46549d88c06e431d0659"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "0e800c180984053eacd330d0c12371ef"
server_key_iv: "d7fd316519a320f8881d37d69bfa2d11"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.180392 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.195822 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.197513 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.197604 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.201047 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42069
I20260924 03:08:50.201097 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42069 every 8 connection(s)
I20260924 03:08:50.201345 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:50.202236 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:50.204275 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: Bootstrap starting.
I20260924 03:08:50.204784 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:50.205044 4521 log.cc:824] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:50.205591 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: No bootstrap required, opened a new log
I20260924 03:08:50.205957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4454
I20260924 03:08:50.206056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:50.206270 392 external_mini_cluster.cc:1428] Setting key 24aa263223ae2f1486f91afaeb095bc5
I20260924 03:08:50.207515 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } }
I20260924 03:08:50.207652 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:50.207688 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ffa0cb11df46549d88c06e431d0659, State: Initialized, Role: FOLLOWER
I20260924 03:08:50.207801 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } }
I20260924 03:08:50.207871 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:50.207903 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:50.207952 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:50.208583 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } }
I20260924 03:08:50.208663 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ffa0cb11df46549d88c06e431d0659; no voters:
I20260924 03:08:50.208856 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:50.208887 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:50.209005 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [term 1 LEADER]: Becoming Leader. State: Replica: 33ffa0cb11df46549d88c06e431d0659, State: Running, Role: LEADER
I20260924 03:08:50.209095 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:50.209081 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } }
I20260924 03:08:50.209375 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ffa0cb11df46549d88c06e431d0659. Latest consensus state: current_term: 1 leader_uuid: "33ffa0cb11df46549d88c06e431d0659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } } }
I20260924 03:08:50.209453 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:50.209384 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ffa0cb11df46549d88c06e431d0659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ffa0cb11df46549d88c06e431d0659" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42069 } } }
I20260924 03:08:50.209594 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659 [sys.catalog]: This master's current role is: LEADER
W20260924 03:08:50.210798 4541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:50.210963 4541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:50.211050 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:50.211227 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:50.215265 4531 catalog_manager.cc:1347] Generated new cluster ID: 6d521f258b1c438da6aeb7e8d91debb1
I20260924 03:08:50.215327 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:50.217314 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.206949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.98.62:42069)
0924 03:08:50.207135 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.207141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.207161 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:08:50.207494 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.207499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.207550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.207642 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.208430 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.209777 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.210887 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.211526 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.211892 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.214055 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.214059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.214060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.214070 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.215866 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.216225 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.216228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.216228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.216327 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.216539 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.216540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.216540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.216751 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.216906 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.216997 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.217132 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260924 03:08:50.228694 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:50.229383 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:50.234516 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: Generated new TSK 0
I20260924 03:08:50.234650 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:50.260257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.335623 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.335796 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.335817 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.336823 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.336851 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.336874 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:50.338114 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:50.338313 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.338471 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.342933 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.342937 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.342995 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.343156 4547 server_base.cc:1034] running on GCE node
I20260924 03:08:50.343454 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.343729 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.344900 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790219330344888 us; error 28 us; skew 500 ppm
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.347362 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:50.347666 4547 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.347709 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.347787 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.348605 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "734fb1aff3ae44629be8233e629aa4a5"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "d792d4014fc4bcaf60a59cbe95d980b5"
server_key_iv: "b39c37a444112970b08ed2a29ceec5a0"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.348886 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "734fb1aff3ae44629be8233e629aa4a5"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "d792d4014fc4bcaf60a59cbe95d980b5"
server_key_iv: "b39c37a444112970b08ed2a29ceec5a0"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.350008 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:50.350647 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.350821 4547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.350891 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "734fb1aff3ae44629be8233e629aa4a5"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "d792d4014fc4bcaf60a59cbe95d980b5"
server_key_iv: "b39c37a444112970b08ed2a29ceec5a0"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.350936 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.369874 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.371868 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.371964 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.372090 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:50.372349 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:50.372380 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.372397 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:50.372437 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.377687 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36891
I20260924 03:08:50.377729 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36891 every 8 connection(s)
I20260924 03:08:50.377931 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:50.384217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4547
I20260924 03:08:50.384320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:50.384485 392 external_mini_cluster.cc:1428] Setting key fdb8fe2b65ee96854a8fb694bff3aa9f
I20260924 03:08:50.385886 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.378535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42069 (local address 127.0.98.1:54741)
0924 03:08:50.378684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.378712 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.379102 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.379274 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.379277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.379486 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:50.379925 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.379930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.380536 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.380537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.381225 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.381226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.381514 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.381514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.381598 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.382065 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:50.382073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:50.383316 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:50.384691 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.384697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.384699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.384936 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.385117 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.385118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.385118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.385218 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.385504 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:50.385507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:50.385596 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.385666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.385761 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 03:08:50.385915 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.378533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54741 (local address 127.0.98.62:42069)
0924 03:08:50.378577 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.378579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.378816 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:08:50.379174 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.379175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.379183 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.379230 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.379998 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.380467 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.381288 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.381456 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.381501 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.383390 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.383391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.383391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.383398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.384629 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.385004 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.385005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.385006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.385054 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.385299 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.385300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.385300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.385440 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.385508 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.385707 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.385789 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:50.386272 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42069
I20260924 03:08:50.386360 4675 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:50.386492 4675 heartbeater.cc:511] Master 127.0.98.62:42069 requested a full tablet report, sending...
I20260924 03:08:50.387046 4485 ts_manager.cc:194] Registered new tserver with Master: 734fb1aff3ae44629be8233e629aa4a5 (127.0.98.1:36891)
I20260924 03:08:50.387928 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54741
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:50.408181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.482280 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.482457 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.482482 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.483636 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.483671 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.483724 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:50.485112 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:50.485329 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.485548 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.490381 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.490356 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.490389 4680 server_base.cc:1034] running on GCE node
W20260924 03:08:50.490365 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.490779 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.491016 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.492168 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790219330492166 us; error 33 us; skew 500 ppm
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.494550 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:50.494884 4680 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.494930 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.495014 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.495925 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "602bbeb0875b4b8781e97e76359ae963"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "76dc3736173fa636f841b0ef0112a302"
server_key_iv: "adee2671d36b4234de4689975e176e83"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.496222 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "602bbeb0875b4b8781e97e76359ae963"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "76dc3736173fa636f841b0ef0112a302"
server_key_iv: "adee2671d36b4234de4689975e176e83"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.497399 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:08:50.498090 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.498265 4680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.498345 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "602bbeb0875b4b8781e97e76359ae963"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "76dc3736173fa636f841b0ef0112a302"
server_key_iv: "adee2671d36b4234de4689975e176e83"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.498410 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.512102 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.514076 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.514250 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.514451 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:50.514745 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:50.514776 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.514796 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:50.514839 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.519881 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45997
I20260924 03:08:50.519910 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45997 every 8 connection(s)
I20260924 03:08:50.520157 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:50.522531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4680
I20260924 03:08:50.522636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:50.522811 392 external_mini_cluster.cc:1428] Setting key 5cf61d1c3d158c1cd26b9ac52b388928
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:50.530426 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.520848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42069 (local address 127.0.98.2:60721)
0924 03:08:50.521025 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.521057 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.521485 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.521717 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.521720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.521986 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:50.522504 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.522509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.523261 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.523262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.524070 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.524073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.524497 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.524499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.524672 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.525106 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:50.525113 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:50.526664 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:50.528624 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.528631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.528633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.528956 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.529211 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.529212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.529229 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.529359 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.529729 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:50.529735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:50.529889 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.530007 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.530167 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260924 03:08:50.530369 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.520863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60721 (local address 127.0.98.62:42069)
0924 03:08:50.520935 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.520937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.521130 (+ 193us) server_negotiation.cc:407] Connection header received
0924 03:08:50.521584 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.521585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.521597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.521657 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.522605 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.523159 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.524156 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.524358 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.524454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.526773 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.526775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.526775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.526783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.528518 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.529058 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.529059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.529059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.529141 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.529439 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.529440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.529441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.529634 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.529738 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.530041 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.530200 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:08:50.531311 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42069
I20260924 03:08:50.531451 4808 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:50.531598 4808 heartbeater.cc:511] Master 127.0.98.62:42069 requested a full tablet report, sending...
I20260924 03:08:50.532168 4485 ts_manager.cc:194] Registered new tserver with Master: 602bbeb0875b4b8781e97e76359ae963 (127.0.98.2:45997)
I20260924 03:08:50.532617 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:50.548110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:50.624603 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.624799 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.624820 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.625938 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.625969 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.625993 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:50.627328 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:50.627481 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.627620 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.632098 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.632136 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.632194 4813 server_base.cc:1034] running on GCE node
W20260924 03:08:50.632164 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.632504 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.632745 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.633881 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790219330633867 us; error 27 us; skew 500 ppm
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:50.636457 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:50.636780 4813 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.636847 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.636986 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.637916 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd43d028273f4897b2df8fb4cce7427a"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "75e56405d917a7cb4a72161c7e9353a5"
server_key_iv: "9305cec8fc3198903a812be089576233"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.638299 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd43d028273f4897b2df8fb4cce7427a"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "75e56405d917a7cb4a72161c7e9353a5"
server_key_iv: "9305cec8fc3198903a812be089576233"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.639479 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:50.640110 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.640267 4813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:50.640354 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "bd43d028273f4897b2df8fb4cce7427a"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "75e56405d917a7cb4a72161c7e9353a5"
server_key_iv: "9305cec8fc3198903a812be089576233"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.640429 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.646703 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.648301 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.648447 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.648643 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:50.648916 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:50.648972 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.649003 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:50.649029 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.654546 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43317
I20260924 03:08:50.654574 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43317 every 8 connection(s)
I20260924 03:08:50.654796 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:08:50 dist-test-slave-jtnz krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:50.662066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4813
I20260924 03:08:50.662170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:50.662436 392 external_mini_cluster.cc:1428] Setting key 5fcf4e2ff33d8de160583c3654b9798f
I20260924 03:08:50.663347 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.655542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49257 (local address 127.0.98.62:42069)
0924 03:08:50.655587 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.655589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.655763 (+ 174us) server_negotiation.cc:407] Connection header received
0924 03:08:50.656171 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.656172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.656181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.656230 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.657002 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.657463 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.658455 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.658630 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.658681 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.660559 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.660560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.660560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.660566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.661697 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.662164 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.662165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.662165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.662268 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.662598 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.662599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.662600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.662754 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.662817 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.663182 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.663253 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:50.663480 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.655517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42069 (local address 127.0.98.3:49257)
0924 03:08:50.655660 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.655687 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.656096 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.656274 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.656277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.656491 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:50.656933 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.656937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.657553 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.657554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.658379 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.658381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.658715 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.658716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.658831 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.659207 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:50.659214 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:50.660466 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:50.661761 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.661765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.661766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.662077 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.662346 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:50.662347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:50.662347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:50.662481 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:50.662846 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:50.662851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:50.663005 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.663145 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.663262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260924 03:08:50.663929 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42069
I20260924 03:08:50.664009 4941 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:50.664103 4941 heartbeater.cc:511] Master 127.0.98.62:42069 requested a full tablet report, sending...
I20260924 03:08:50.664383 4485 ts_manager.cc:194] Registered new tserver with Master: bd43d028273f4897b2df8fb4cce7427a (127.0.98.3:43317)
I20260924 03:08:50.664716 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49257
I20260924 03:08:50.674204 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:50.680409 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.675442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.0.98.62:42069)
0924 03:08:50.675494 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.675496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.675587 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:08:50.675623 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.675623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.675631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.675679 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.676181 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.676609 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.677232 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.677389 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.677432 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.678428 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.678429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.678430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:50.678435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.679520 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.679807 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.679807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.679808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.679850 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:50.680034 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:50.680034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:50.680034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:50.680156 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.680220 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.680312 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.680359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:08:50.681881 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:50.683437 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:50.689102 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.0.98.3:43317)
0924 03:08:50.685763 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.685767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.685780 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:08:50.685814 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.685816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.685830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.685904 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:50.686041 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.686883 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.688171 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.688705 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.688799 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.688871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.689001 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.689034 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":82,"threads_started":1}
I20260924 03:08:50.689106 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43317 (local address 127.0.0.1:43214)
0924 03:08:50.685539 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.685575 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.685636 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.685908 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.685910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.685912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:50.685967 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.685969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.686979 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.686980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.688090 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.688092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.688803 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.688805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.688900 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.688911 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.688978 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.689028 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":46,"threads_started":1}
I20260924 03:08:50.689419 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36891 (local address 127.0.0.1:39600)
0924 03:08:50.685709 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.685719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.685775 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.686036 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.686037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.686038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:50.686084 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.686085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.687323 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.687324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.688299 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.688300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.689176 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.689177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.689256 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.689259 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.689311 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.689348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":7,"thread_start_us":42,"threads_started":1}
I20260924 03:08:50.689450 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.98.1:36891)
0924 03:08:50.685646 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.685649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.685830 (+ 181us) server_negotiation.cc:407] Connection header received
0924 03:08:50.685866 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.685868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.685881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.685948 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:50.686163 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.687227 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.688370 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.689045 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.689102 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.689147 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.689326 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.689374 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260924 03:08:50.689682 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 (DEFAULT_TABLE table=test-table [id=e391357feb5c4062a86136ba1039c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:50.689952 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7e40d4bf714a05b83cb72cf3944098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:50.690094 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45997 (local address 127.0.0.1:34456)
0924 03:08:50.685735 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.685745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.685797 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.686031 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.686032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.686034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:50.686084 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.686085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.687430 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.687431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.688152 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.688154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.689140 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.689142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.689439 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.689443 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.689941 (+ 498us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.689995 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":71,"threads_started":1}
I20260924 03:08:50.690131 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 (DEFAULT_TABLE table=test-table [id=e391357feb5c4062a86136ba1039c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:50.690389 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.685585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.0.98.2:45997)
0924 03:08:50.685751 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.685755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.685787 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:08:50.685862 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.685864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.685878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.685948 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:50.686161 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.687337 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.688235 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.688967 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.689048 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.689105 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.690271 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.690319 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260924 03:08:50.690464 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7e40d4bf714a05b83cb72cf3944098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:50.690932 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 (DEFAULT_TABLE table=test-table [id=e391357feb5c4062a86136ba1039c3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:50.691268 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7e40d4bf714a05b83cb72cf3944098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:50.693008 4956 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Bootstrap starting.
I20260924 03:08:50.692770 4955 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: Bootstrap starting.
I20260924 03:08:50.693564 4955 tablet_bootstrap.cc:654] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:50.693569 4956 tablet_bootstrap.cc:654] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:50.694053 4955 log.cc:824] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:50.694139 4956 log.cc:824] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:50.694434 4957 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Bootstrap starting.
I20260924 03:08:50.694717 4955 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: No bootstrap required, opened a new log
I20260924 03:08:50.694795 4955 ts_tablet_manager.cc:1397] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:08:50.694823 4956 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: No bootstrap required, opened a new log
I20260924 03:08:50.694882 4956 ts_tablet_manager.cc:1397] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:50.694999 4957 tablet_bootstrap.cc:654] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:50.695530 4957 log.cc:824] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:50.696161 4957 tablet_bootstrap.cc:492] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: No bootstrap required, opened a new log
I20260924 03:08:50.696228 4957 ts_tablet_manager.cc:1397] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:50.696434 4955 raft_consensus.cc:348] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.696563 4955 raft_consensus.cc:374] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:50.696600 4955 raft_consensus.cc:729] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734fb1aff3ae44629be8233e629aa4a5, State: Initialized, Role: FOLLOWER
I20260924 03:08:50.696713 4955 consensus_queue.cc:260] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.696940 4955 ts_tablet_manager.cc:1428] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:08:50.696882 4956 raft_consensus.cc:348] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.697032 4956 raft_consensus.cc:374] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:50.697062 4675 heartbeater.cc:503] Master 127.0.98.62:42069 was elected leader, sending a full tablet report...
I20260924 03:08:50.697072 4956 raft_consensus.cc:729] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd43d028273f4897b2df8fb4cce7427a, State: Initialized, Role: FOLLOWER
I20260924 03:08:50.697160 4956 consensus_queue.cc:260] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.697387 4956 ts_tablet_manager.cc:1428] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:08:50.697639 4941 heartbeater.cc:503] Master 127.0.98.62:42069 was elected leader, sending a full tablet report...
I20260924 03:08:50.698112 4957 raft_consensus.cc:348] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.698287 4957 raft_consensus.cc:374] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:50.698324 4957 raft_consensus.cc:729] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602bbeb0875b4b8781e97e76359ae963, State: Initialized, Role: FOLLOWER
I20260924 03:08:50.698421 4957 consensus_queue.cc:260] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.698663 4957 ts_tablet_manager.cc:1428] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:08:50.698684 4808 heartbeater.cc:503] Master 127.0.98.62:42069 was elected leader, sending a full tablet report...
W20260924 03:08:50.770664 4809 tablet.cc:2369] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:50.816295 4963 raft_consensus.cc:482] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:50.816387 4963 raft_consensus.cc:504] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.816715 4963 leader_election.cc:288] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734fb1aff3ae44629be8233e629aa4a5 (127.0.98.1:36891), bd43d028273f4897b2df8fb4cce7427a (127.0.98.3:43317)
I20260924 03:08:50.819378 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.816987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39903 (local address 127.0.98.1:36891)
0924 03:08:50.817025 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.817026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.817072 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:08:50.817136 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.817137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.817147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.817199 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:50.817368 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.817763 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.818678 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.819028 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.819078 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.819086 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.819272 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.819317 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:08:50.819388 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.816886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36891 (local address 127.0.98.2:39903)
0924 03:08:50.817018 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.817027 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.817079 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.817211 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.817212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.817213 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:50.817311 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.817313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.817859 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.817859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.818590 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.818591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.819109 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.819110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.819181 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.819184 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.819237 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.819282 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260924 03:08:50.819456 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.816946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41815 (local address 127.0.98.3:43317)
0924 03:08:50.816987 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.816990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.817000 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:08:50.817065 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.817065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.817076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.817124 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:50.817280 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.817738 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.818680 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.819088 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.819156 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.819165 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.819349 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.819385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:50.819650 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.816873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43317 (local address 127.0.98.2:41815)
0924 03:08:50.816932 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.816945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:50.817002 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:50.817137 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:50.817138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:50.817139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:50.817207 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.817209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.817812 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.817814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.818589 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:50.818591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.819168 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.819169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:50.819248 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.819251 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:50.819323 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:08:50.819359 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":23}
I20260924 03:08:50.819667 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7e40d4bf714a05b83cb72cf3944098" candidate_uuid: "602bbeb0875b4b8781e97e76359ae963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734fb1aff3ae44629be8233e629aa4a5" is_pre_election: true
I20260924 03:08:50.819813 4629 raft_consensus.cc:2454] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 602bbeb0875b4b8781e97e76359ae963 in term 0.
I20260924 03:08:50.820027 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7e40d4bf714a05b83cb72cf3944098" candidate_uuid: "602bbeb0875b4b8781e97e76359ae963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd43d028273f4897b2df8fb4cce7427a" is_pre_election: true
I20260924 03:08:50.820065 4695 leader_election.cc:302] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 602bbeb0875b4b8781e97e76359ae963, 734fb1aff3ae44629be8233e629aa4a5; no voters:
I20260924 03:08:50.820173 4895 raft_consensus.cc:2454] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 602bbeb0875b4b8781e97e76359ae963 in term 0.
I20260924 03:08:50.820194 4963 raft_consensus.cc:2781] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:50.820266 4963 raft_consensus.cc:482] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:50.820290 4963 raft_consensus.cc:3037] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:50.820930 4963 raft_consensus.cc:504] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.821071 4963 leader_election.cc:288] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [CANDIDATE]: Term 1 election: Requested vote from peers 734fb1aff3ae44629be8233e629aa4a5 (127.0.98.1:36891), bd43d028273f4897b2df8fb4cce7427a (127.0.98.3:43317)
I20260924 03:08:50.821197 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7e40d4bf714a05b83cb72cf3944098" candidate_uuid: "602bbeb0875b4b8781e97e76359ae963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734fb1aff3ae44629be8233e629aa4a5"
I20260924 03:08:50.821211 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7e40d4bf714a05b83cb72cf3944098" candidate_uuid: "602bbeb0875b4b8781e97e76359ae963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd43d028273f4897b2df8fb4cce7427a"
I20260924 03:08:50.821269 4629 raft_consensus.cc:3037] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:50.821295 4895 raft_consensus.cc:3037] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:50.821926 4629 raft_consensus.cc:2454] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 602bbeb0875b4b8781e97e76359ae963 in term 1.
I20260924 03:08:50.822055 4895 raft_consensus.cc:2454] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 602bbeb0875b4b8781e97e76359ae963 in term 1.
I20260924 03:08:50.822098 4695 leader_election.cc:302] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 602bbeb0875b4b8781e97e76359ae963, 734fb1aff3ae44629be8233e629aa4a5; no voters:
I20260924 03:08:50.822207 4963 raft_consensus.cc:2781] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:50.822346 4963 raft_consensus.cc:686] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 1 LEADER]: Becoming Leader. State: Replica: 602bbeb0875b4b8781e97e76359ae963, State: Running, Role: LEADER
I20260924 03:08:50.822458 4963 consensus_queue.cc:237] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } }
I20260924 03:08:50.823203 4485 catalog_manager.cc:5495] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 reported cstate change: term changed from 0 to 1, leader changed from <none> to 602bbeb0875b4b8781e97e76359ae963 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "602bbeb0875b4b8781e97e76359ae963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "602bbeb0875b4b8781e97e76359ae963" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 } health_report { overall_health: UNKNOWN } } }
I20260924 03:08:50.833700 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.831923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.0.98.2:45997)
0924 03:08:50.831956 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.831958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.831967 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:50.832032 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.832032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.832045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.832094 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:50.832204 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.832573 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.833173 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:50.833321 (+ 148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:50.833362 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:50.833476 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:50.833554 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:50.833604 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.833636 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.833649 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:50.835381 4629 raft_consensus.cc:1261] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 1 FOLLOWER]: Refusing update from remote peer 602bbeb0875b4b8781e97e76359ae963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:50.835381 4895 raft_consensus.cc:1261] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 1 FOLLOWER]: Refusing update from remote peer 602bbeb0875b4b8781e97e76359ae963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:50.835686 4963 consensus_queue.cc:1035] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734fb1aff3ae44629be8233e629aa4a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:50.835820 4965 consensus_queue.cc:1035] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd43d028273f4897b2df8fb4cce7427a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:50.837625 4968 mvcc.cc:205] Tried to move back new op lower bound from 7332738379099348992 to 7332738379049373696. Current Snapshot: MvccSnapshot[applied={T|T < 7332738379099348992}]
I20260924 03:08:50.837918 4970 mvcc.cc:205] Tried to move back new op lower bound from 7332738379099348992 to 7332738379049373696. Current Snapshot: MvccSnapshot[applied={T|T < 7332738379099348992}]
I20260924 03:08:50.837967 4969 mvcc.cc:205] Tried to move back new op lower bound from 7332738379099348992 to 7332738379049373696. Current Snapshot: MvccSnapshot[applied={T|T < 7332738379099348992}]
I20260924 03:08:50.839849 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:50.839936 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:50.840593 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33ffa0cb11df46549d88c06e431d0659: Sending DeleteTablet for 3 replicas of tablet 2d7e40d4bf714a05b83cb72cf3944098
I20260924 03:08:50.840965 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43214
I20260924 03:08:50.841171 4978 tablet_replica.cc:330] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: stopping tablet replica
I20260924 03:08:50.841174 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39600
I20260924 03:08:50.841280 4978 raft_consensus.cc:2229] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:50.841382 4978 raft_consensus.cc:2258] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:50.841439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4547
I20260924 03:08:50.841405 4979 tablet_replica.cc:330] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5: stopping tablet replica
I20260924 03:08:50.841542 4979 raft_consensus.cc:2229] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:50.841624 4979 raft_consensus.cc:2258] T 2d7e40d4bf714a05b83cb72cf3944098 P 734fb1aff3ae44629be8233e629aa4a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:50.842896 4978 ts_tablet_manager.cc:1905] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:50.843189 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7e40d4bf714a05b83cb72cf3944098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34456
I20260924 03:08:50.843360 4980 tablet_replica.cc:330] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: stopping tablet replica
I20260924 03:08:50.843442 4980 raft_consensus.cc:2229] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:50.843537 4980 raft_consensus.cc:2258] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:50.843894 4980 ts_tablet_manager.cc:1905] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:50.844923 4978 ts_tablet_manager.cc:1918] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:50.845042 4978 log.cc:1196] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/2d7e40d4bf714a05b83cb72cf3944098
I20260924 03:08:50.845386 4978 ts_tablet_manager.cc:1939] T 2d7e40d4bf714a05b83cb72cf3944098 P bd43d028273f4897b2df8fb4cce7427a: Deleting consensus metadata
I20260924 03:08:50.845491 4980 ts_tablet_manager.cc:1918] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:50.845571 4980 log.cc:1196] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/2d7e40d4bf714a05b83cb72cf3944098
I20260924 03:08:50.845825 4980 ts_tablet_manager.cc:1939] T 2d7e40d4bf714a05b83cb72cf3944098 P 602bbeb0875b4b8781e97e76359ae963: Deleting consensus metadata
I20260924 03:08:50.845916 4470 catalog_manager.cc:4841] TS bd43d028273f4897b2df8fb4cce7427a (127.0.98.3:43317): tablet 2d7e40d4bf714a05b83cb72cf3944098 (table test-table [id=e391357feb5c4062a86136ba1039c3e4]) successfully deleted
W20260924 03:08:50.848438 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36891 (error 108)
I20260924 03:08:50.848631 4469 catalog_manager.cc:4841] TS 602bbeb0875b4b8781e97e76359ae963 (127.0.98.2:45997): tablet 2d7e40d4bf714a05b83cb72cf3944098 (table test-table [id=e391357feb5c4062a86136ba1039c3e4]) successfully deleted
I20260924 03:08:50.848956 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.848804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36891 (local address 127.0.0.1:39614)
0924 03:08:50.848873 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:50.848925 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:08:50.848953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4680
W20260924 03:08:50.850605 4469 catalog_manager.cc:4584] TS 734fb1aff3ae44629be8233e629aa4a5 (127.0.98.1:36891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d7e40d4bf714a05b83cb72cf3944098: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36891: connect: Connection refused (error 111)
I20260924 03:08:50.853562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4813
I20260924 03:08:50.857628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4454
2026-09-24T03:08:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1867 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T03:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:50Z Disabled control of system clock
I20260924 03:08:50.883049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:08:50.957619 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:50.957793 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:50.957810 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:50.958797 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:50.958847 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:50.958860 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:50.958873 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:50.958884 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:50.960094 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:50.960253 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:50.960450 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:50.964751 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.964793 4985 server_base.cc:1034] running on GCE node
W20260924 03:08:50.964797 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:50.964762 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:50.965147 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:50.965389 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:50.966522 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790219330966511 us; error 26 us; skew 500 ppm
I20260924 03:08:50.967681 4985 webserver.cc:460] Webserver started at http://127.0.98.62:45551/ using document root <none> and password file <none>
I20260924 03:08:50.967892 4985 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:50.967935 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:50.968082 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:50.969044 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "9c975b93344e48b6836c6b4fc38a64a6"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "e5ddea5eba4293ac4a637fe5af893d2e"
server_key_iv: "3231df1668ff2b3ed46c041a81a36d7b"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.969377 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c975b93344e48b6836c6b4fc38a64a6"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "e5ddea5eba4293ac4a637fe5af893d2e"
server_key_iv: "3231df1668ff2b3ed46c041a81a36d7b"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.970650 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:50.971288 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:50.971438 4985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:50.971540 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9c975b93344e48b6836c6b4fc38a64a6"
format_stamp: "Formatted at 2026-09-24 03:08:50 on dist-test-slave-jtnz"
server_key: "e5ddea5eba4293ac4a637fe5af893d2e"
server_key_iv: "3231df1668ff2b3ed46c041a81a36d7b"
server_key_version: "encryptionkey@0"
I20260924 03:08:50.971608 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:50.974684 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:50.974916 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:50.975049 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:50.978667 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43743
I20260924 03:08:50.978694 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43743 every 8 connection(s)
I20260924 03:08:50.978995 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:50.979553 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:50.981956 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: Bootstrap starting.
I20260924 03:08:50.982501 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:50.982795 5052 log.cc:824] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:50.983392 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: No bootstrap required, opened a new log
I20260924 03:08:50.984830 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } }
I20260924 03:08:50.984946 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:50.984965 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c975b93344e48b6836c6b4fc38a64a6, State: Initialized, Role: FOLLOWER
I20260924 03:08:50.985080 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } }
I20260924 03:08:50.985155 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:50.985176 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:50.985201 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:50.985659 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } }
I20260924 03:08:50.985734 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c975b93344e48b6836c6b4fc38a64a6; no voters:
I20260924 03:08:50.985841 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:50.985898 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:50.986096 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:50.986112 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c975b93344e48b6836c6b4fc38a64a6, State: Running, Role: LEADER
I20260924 03:08:50.986348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 4985
I20260924 03:08:50.986451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:50.986634 392 external_mini_cluster.cc:1428] Setting key cff7c0749068b986604955cf85a31704
I20260924 03:08:50.986805 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } }
I20260924 03:08:50.987411 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c975b93344e48b6836c6b4fc38a64a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } } }
I20260924 03:08:50.987413 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c975b93344e48b6836c6b4fc38a64a6. Latest consensus state: current_term: 1 leader_uuid: "9c975b93344e48b6836c6b4fc38a64a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c975b93344e48b6836c6b4fc38a64a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43743 } } }
I20260924 03:08:50.987527 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:50.987543 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6 [sys.catalog]: This master's current role is: LEADER
W20260924 03:08:50.988243 5072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:50.988333 5072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:50.988426 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:50.988612 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:50.989384 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:50.987718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.0.98.62:43743)
0924 03:08:50.987979 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:50.987988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:50.988019 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:08:50.988100 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:50.988104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:50.988133 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:50.988267 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:50.988878 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:50.988881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:50.988882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:50.988886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:50.988931 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:50.988943 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:50.989016 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:50.989088 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:08:50.989230 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":57,"threads_started":1}
I20260924 03:08:50.989995 5071 catalog_manager.cc:1347] Generated new cluster ID: 498752739e7f47a59370edb335f81a8d
I20260924 03:08:50.990070 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:51.018736 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:51.018918 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:51.025641 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: Generated new TSK 0
I20260924 03:08:51.025827 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:51.027695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--builtin_ntp_servers=127.0.98.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:08:51.105566 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.105727 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.105746 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.106819 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.106854 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.106876 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:51.108078 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.108336 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.108635 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.113818 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.113799 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.113982 5076 server_base.cc:1034] running on GCE node
W20260924 03:08:51.113801 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.114292 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.114567 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.115818 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790219331115805 us; error 27 us; skew 500 ppm
I20260924 03:08:51.116981 5076 webserver.cc:460] Webserver started at http://127.0.98.1:33851/ using document root <none> and password file <none>
I20260924 03:08:51.117204 5076 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.117266 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.117410 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.118405 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c9657286ba54fa0ae49fb432433cc34"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "41bf6574c3390dda557cc11b7195af3e"
server_key_iv: "5f4490cdcce1a82fac8db7f03ee62c2b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.118724 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c9657286ba54fa0ae49fb432433cc34"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "41bf6574c3390dda557cc11b7195af3e"
server_key_iv: "5f4490cdcce1a82fac8db7f03ee62c2b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.119973 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:51.120647 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.120801 5076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:51.120893 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "8c9657286ba54fa0ae49fb432433cc34"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "41bf6574c3390dda557cc11b7195af3e"
server_key_iv: "5f4490cdcce1a82fac8db7f03ee62c2b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.120975 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.124100 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.124327 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.124454 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.124652 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:51.124912 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:51.124969 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.125003 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:51.125025 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.130111 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35315
I20260924 03:08:51.130158 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35315 every 8 connection(s)
I20260924 03:08:51.130429 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:51.132203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5076
I20260924 03:08:51.132331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:51.132526 392 external_mini_cluster.cc:1428] Setting key 6b954f5ee91327f07f56eb315bbf8514
I20260924 03:08:51.133065 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.131084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43529 (local address 127.0.98.62:43743)
0924 03:08:51.131123 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.131125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.131321 (+ 196us) server_negotiation.cc:407] Connection header received
0924 03:08:51.131684 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.131685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.131690 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.131733 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.132856 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.132857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.132857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.132861 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.132875 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.132879 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.132926 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.132988 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.132998 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:51.133227 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.131072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43743 (local address 127.0.98.1:43529)
0924 03:08:51.131219 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.131246 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.131611 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.131784 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.131786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.131793 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.132714 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.132715 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.132750 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.132772 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.132925 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.132929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.132931 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.132997 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.133049 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260924 03:08:51.133440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--builtin_ntp_servers=127.0.98.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:08:51.134922 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43743
I20260924 03:08:51.135015 5204 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:51.135156 5204 heartbeater.cc:511] Master 127.0.98.62:43743 requested a full tablet report, sending...
I20260924 03:08:51.135588 5016 ts_manager.cc:194] Registered new tserver with Master: 8c9657286ba54fa0ae49fb432433cc34 (127.0.98.1:35315)
W20260924 03:08:51.207819 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.207983 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.208002 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.209069 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.209105 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.209131 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:51.210394 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.210592 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.210839 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.215111 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.215140 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.215124 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.215394 5207 server_base.cc:1034] running on GCE node
I20260924 03:08:51.215579 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.215792 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.216921 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790219331216900 us; error 36 us; skew 500 ppm
I20260924 03:08:51.218096 5207 webserver.cc:460] Webserver started at http://127.0.98.2:40833/ using document root <none> and password file <none>
I20260924 03:08:51.218334 5207 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.218379 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.218501 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.219460 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad546fd766a1443ebb1e6330a821a6b8"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "2ec4710fa4ae48068d30c684ade4d101"
server_key_iv: "115d83370876be56bd5a09855c027a9c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.219794 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad546fd766a1443ebb1e6330a821a6b8"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "2ec4710fa4ae48068d30c684ade4d101"
server_key_iv: "115d83370876be56bd5a09855c027a9c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.220988 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:51.221604 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.221751 5207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:51.221840 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "ad546fd766a1443ebb1e6330a821a6b8"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "2ec4710fa4ae48068d30c684ade4d101"
server_key_iv: "115d83370876be56bd5a09855c027a9c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.221927 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.225281 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.225525 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.225683 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.225885 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:51.226153 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:51.226236 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.226274 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:51.226301 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.231799 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39405
I20260924 03:08:51.231832 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39405 every 8 connection(s)
I20260924 03:08:51.232122 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:51.234535 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.232745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43611 (local address 127.0.98.62:43743)
0924 03:08:51.232817 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.232819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.232976 (+ 157us) server_negotiation.cc:407] Connection header received
0924 03:08:51.233325 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.233326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.233330 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.233377 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.234334 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.234335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.234336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.234340 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.234353 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.234357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.234406 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.234460 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.234470 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:08:51.234688 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.232714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43743 (local address 127.0.98.2:43611)
0924 03:08:51.232877 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.232904 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.233254 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.233418 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.233421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.233428 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.234163 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.234163 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.234229 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.234257 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.234405 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.234407 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.234409 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.234466 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.234521 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 03:08:51.235105 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43743
I20260924 03:08:51.235186 5335 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:51.235278 5335 heartbeater.cc:511] Master 127.0.98.62:43743 requested a full tablet report, sending...
I20260924 03:08:51.235569 5016 ts_manager.cc:194] Registered new tserver with Master: ad546fd766a1443ebb1e6330a821a6b8 (127.0.98.2:39405)
I20260924 03:08:51.237854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5207
I20260924 03:08:51.237962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:51.238117 392 external_mini_cluster.cc:1428] Setting key 04ee5b258e84622ca71aecae87cefb2b
I20260924 03:08:51.239043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--builtin_ntp_servers=127.0.98.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:08:51.312508 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.312674 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.312693 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.313716 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.313751 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.313799 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:51.315057 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.315270 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.315506 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.319758 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.319792 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.319813 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.319911 5338 server_base.cc:1034] running on GCE node
I20260924 03:08:51.320098 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.320279 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.321398 5338 hybrid_clock.cc:648] HybridClock initialized: now 1790219331321381 us; error 36 us; skew 500 ppm
I20260924 03:08:51.322615 5338 webserver.cc:460] Webserver started at http://127.0.98.3:35701/ using document root <none> and password file <none>
I20260924 03:08:51.322850 5338 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.322894 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.322978 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.323887 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "51e337b9691b4ec2840c47cfd9030d11"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "b147040977521586b0e3afbb0822b1a3"
server_key_iv: "796f49b98bbff38911dec69299fe1e44"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.324144 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51e337b9691b4ec2840c47cfd9030d11"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "b147040977521586b0e3afbb0822b1a3"
server_key_iv: "796f49b98bbff38911dec69299fe1e44"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.325459 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:51.326066 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.326216 5338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:51.326299 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "51e337b9691b4ec2840c47cfd9030d11"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "b147040977521586b0e3afbb0822b1a3"
server_key_iv: "796f49b98bbff38911dec69299fe1e44"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.326345 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.329152 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.329332 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.329418 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.329550 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:51.329779 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:51.329807 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.329825 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:51.329838 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.334813 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44481
I20260924 03:08:51.334905 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44481 every 8 connection(s)
I20260924 03:08:51.335157 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:51.337695 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.335774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43351 (local address 127.0.98.62:43743)
0924 03:08:51.335865 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.335868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.336031 (+ 163us) server_negotiation.cc:407] Connection header received
0924 03:08:51.336454 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.336455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.336461 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.336511 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.337464 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.337465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.337466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.337470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.337485 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.337489 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.337533 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.337618 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.337632 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:08:51.337852 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.335740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43743 (local address 127.0.98.3:43351)
0924 03:08:51.335934 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.335960 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.336345 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.336569 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.336573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.336587 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.337337 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.337337 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.337360 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.337382 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.337538 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.337542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.337545 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.337626 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.337676 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260924 03:08:51.338335 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43743
I20260924 03:08:51.338398 5466 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:51.338486 5466 heartbeater.cc:511] Master 127.0.98.62:43743 requested a full tablet report, sending...
I20260924 03:08:51.338757 5016 ts_manager.cc:194] Registered new tserver with Master: 51e337b9691b4ec2840c47cfd9030d11 (127.0.98.3:44481)
I20260924 03:08:51.343062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5338
I20260924 03:08:51.343155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:51.343274 392 external_mini_cluster.cc:1428] Setting key 9b6d2e235d783fac9ac9859122089b89
I20260924 03:08:51.344007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:08:51.346177 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.345199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.0.98.62:43743)
0924 03:08:51.345235 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.345237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.345365 (+ 128us) server_negotiation.cc:407] Connection header received
0924 03:08:51.345409 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.345409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.345414 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.345454 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.345993 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.345994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.345994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.345996 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.346004 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.346007 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.346039 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.346077 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.346087 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:51.347448 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:51.348817 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:51.352319 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.350698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39405 (local address 127.0.0.1:53592)
0924 03:08:51.350956 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.351001 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.351099 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.351567 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.351568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.351570 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.351918 (+ 348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.351918 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.351925 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.351936 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.352144 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.352144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.352145 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.352200 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.352213 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":42,"threads_started":1}
I20260924 03:08:51.352322 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.350902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44481 (local address 127.0.0.1:52574)
0924 03:08:51.351020 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.351028 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.351084 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.351320 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.351326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.351332 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.351775 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.351775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.351790 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.351812 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.352131 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.352137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.352139 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.352194 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.352207 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260924 03:08:51.352573 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.350777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.2:39405)
0924 03:08:51.350930 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.350934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.351079 (+ 145us) server_negotiation.cc:407] Connection header received
0924 03:08:51.351179 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.351184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.351194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.351246 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.351998 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.352008 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.352009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.352013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.352054 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.352073 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.352151 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.352463 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.352480 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260924 03:08:51.352897 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.351102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35315 (local address 127.0.0.1:42596)
0924 03:08:51.351698 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.351704 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.351765 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.352168 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.352169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.352171 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.352559 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.352559 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.352565 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.352576 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.352774 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.352775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.352776 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.352833 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.352845 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":62,"threads_started":1}
I20260924 03:08:51.352912 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.350961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.98.3:44481)
0924 03:08:51.351184 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.351187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.351195 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:51.351213 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.351217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.351224 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.351316 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.351872 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.351878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.351878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.352007 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.352041 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.352061 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.352809 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.352833 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.352848 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":54,"threads_started":1}
I20260924 03:08:51.352922 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.351188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.0.98.1:35315)
0924 03:08:51.351575 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.351579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.351772 (+ 193us) server_negotiation.cc:407] Connection header received
0924 03:08:51.351822 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.351825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.351835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.351902 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.352647 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.352653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.352654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.352658 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.352686 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.352711 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.352767 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.352839 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.352852 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":101,"threads_started":1}
I20260924 03:08:51.353195 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 9eba2b49934b495c903a8856818bc6e2 (DEFAULT_TABLE table=test-table [id=346810cebaca44b59cd619f2fc30388c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:51.353518 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eba2b49934b495c903a8856818bc6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.353477 5400 tablet_service.cc:1467] Processing CreateTablet for tablet 9eba2b49934b495c903a8856818bc6e2 (DEFAULT_TABLE table=test-table [id=346810cebaca44b59cd619f2fc30388c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:51.353677 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eba2b49934b495c903a8856818bc6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.353682 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 9eba2b49934b495c903a8856818bc6e2 (DEFAULT_TABLE table=test-table [id=346810cebaca44b59cd619f2fc30388c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:51.353961 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eba2b49934b495c903a8856818bc6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.355816 5480 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Bootstrap starting.
I20260924 03:08:51.356365 5480 tablet_bootstrap.cc:654] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.356685 5481 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Bootstrap starting.
I20260924 03:08:51.356621 5480 log.cc:824] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.357050 5482 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Bootstrap starting.
I20260924 03:08:51.357287 5481 tablet_bootstrap.cc:654] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.357589 5481 log.cc:824] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.357611 5480 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: No bootstrap required, opened a new log
I20260924 03:08:51.357640 5482 tablet_bootstrap.cc:654] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.357671 5480 ts_tablet_manager.cc:1397] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:51.357991 5482 log.cc:824] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.358369 5481 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: No bootstrap required, opened a new log
I20260924 03:08:51.358434 5481 ts_tablet_manager.cc:1397] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:51.358651 5482 tablet_bootstrap.cc:492] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: No bootstrap required, opened a new log
I20260924 03:08:51.358726 5482 ts_tablet_manager.cc:1397] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:08:51.360162 5480 raft_consensus.cc:348] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.360337 5480 raft_consensus.cc:374] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.360375 5480 raft_consensus.cc:729] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e337b9691b4ec2840c47cfd9030d11, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.360526 5480 consensus_queue.cc:260] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.360781 5480 ts_tablet_manager.cc:1428] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:08:51.360757 5482 raft_consensus.cc:348] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.360886 5466 heartbeater.cc:503] Master 127.0.98.62:43743 was elected leader, sending a full tablet report...
I20260924 03:08:51.360911 5482 raft_consensus.cc:374] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.360965 5482 raft_consensus.cc:729] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9657286ba54fa0ae49fb432433cc34, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.360957 5481 raft_consensus.cc:348] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.361099 5481 raft_consensus.cc:374] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.361076 5482 consensus_queue.cc:260] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.361178 5481 raft_consensus.cc:729] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad546fd766a1443ebb1e6330a821a6b8, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.361311 5481 consensus_queue.cc:260] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.361366 5482 ts_tablet_manager.cc:1428] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:08:51.361377 5204 heartbeater.cc:503] Master 127.0.98.62:43743 was elected leader, sending a full tablet report...
I20260924 03:08:51.361574 5481 ts_tablet_manager.cc:1428] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:08:51.361763 5335 heartbeater.cc:503] Master 127.0.98.62:43743 was elected leader, sending a full tablet report...
W20260924 03:08:51.380869 5205 tablet.cc:2369] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:51.396524 5488 raft_consensus.cc:482] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:51.396589 5488 raft_consensus.cc:504] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.396842 5488 leader_election.cc:288] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51e337b9691b4ec2840c47cfd9030d11 (127.0.98.3:44481), 8c9657286ba54fa0ae49fb432433cc34 (127.0.98.1:35315)
I20260924 03:08:51.397881 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.397032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49833 (local address 127.0.98.3:44481)
0924 03:08:51.397105 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.397107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.397114 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:51.397136 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.397136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.397140 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.397195 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.397703 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.397704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.397704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.397708 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.397719 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.397723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.397773 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.397815 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.397827 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:08:51.397881 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.396979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44481 (local address 127.0.98.2:49833)
0924 03:08:51.397033 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.397043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.397086 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.397205 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.397206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.397208 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.397619 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.397620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.397626 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.397634 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.397773 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.397773 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.397774 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.397815 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.397827 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:08:51.398102 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.397219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50687 (local address 127.0.98.1:35315)
0924 03:08:51.397254 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.397256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.397329 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:08:51.397389 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.397390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.397392 (+ 2us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.397425 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.397883 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.397884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.397885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.397889 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.397900 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.397904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.397949 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.398041 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.398052 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:51.398109 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eba2b49934b495c903a8856818bc6e2" candidate_uuid: "ad546fd766a1443ebb1e6330a821a6b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e337b9691b4ec2840c47cfd9030d11" is_pre_election: true
I20260924 03:08:51.398239 5420 raft_consensus.cc:2454] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad546fd766a1443ebb1e6330a821a6b8 in term 0.
I20260924 03:08:51.398301 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.397146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35315 (local address 127.0.98.2:50687)
0924 03:08:51.397269 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.397278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.397338 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.397431 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.397432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.397434 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:51.397790 (+ 356us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:08:51.397790 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:51.397796 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:08:51.397807 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:51.397975 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:51.397976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:51.397977 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:08:51.398202 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 03:08:51.398216 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 03:08:51.398468 5224 leader_election.cc:302] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e337b9691b4ec2840c47cfd9030d11, ad546fd766a1443ebb1e6330a821a6b8; no voters:
I20260924 03:08:51.398564 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eba2b49934b495c903a8856818bc6e2" candidate_uuid: "ad546fd766a1443ebb1e6330a821a6b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9657286ba54fa0ae49fb432433cc34" is_pre_election: true
I20260924 03:08:51.398622 5488 raft_consensus.cc:2781] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:51.398661 5158 raft_consensus.cc:2454] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad546fd766a1443ebb1e6330a821a6b8 in term 0.
I20260924 03:08:51.398681 5488 raft_consensus.cc:482] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:51.398712 5488 raft_consensus.cc:3037] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.399515 5488 raft_consensus.cc:504] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.399643 5488 leader_election.cc:288] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [CANDIDATE]: Term 1 election: Requested vote from peers 51e337b9691b4ec2840c47cfd9030d11 (127.0.98.3:44481), 8c9657286ba54fa0ae49fb432433cc34 (127.0.98.1:35315)
I20260924 03:08:51.399784 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eba2b49934b495c903a8856818bc6e2" candidate_uuid: "ad546fd766a1443ebb1e6330a821a6b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e337b9691b4ec2840c47cfd9030d11"
I20260924 03:08:51.399798 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eba2b49934b495c903a8856818bc6e2" candidate_uuid: "ad546fd766a1443ebb1e6330a821a6b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9657286ba54fa0ae49fb432433cc34"
I20260924 03:08:51.399866 5420 raft_consensus.cc:3037] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.399864 5158 raft_consensus.cc:3037] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.400458 5158 raft_consensus.cc:2454] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad546fd766a1443ebb1e6330a821a6b8 in term 1.
I20260924 03:08:51.400535 5420 raft_consensus.cc:2454] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad546fd766a1443ebb1e6330a821a6b8 in term 1.
I20260924 03:08:51.400605 5224 leader_election.cc:302] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9657286ba54fa0ae49fb432433cc34, ad546fd766a1443ebb1e6330a821a6b8; no voters:
I20260924 03:08:51.400698 5488 raft_consensus.cc:2781] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:51.400835 5488 raft_consensus.cc:686] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 1 LEADER]: Becoming Leader. State: Replica: ad546fd766a1443ebb1e6330a821a6b8, State: Running, Role: LEADER
I20260924 03:08:51.400925 5488 consensus_queue.cc:237] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } }
I20260924 03:08:51.401600 5015 catalog_manager.cc:5495] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad546fd766a1443ebb1e6330a821a6b8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad546fd766a1443ebb1e6330a821a6b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad546fd766a1443ebb1e6330a821a6b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 } health_report { overall_health: UNKNOWN } } }
I20260924 03:08:51.413331 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.412487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.98.2:39405)
0924 03:08:51.412546 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.412548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.412590 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:08:51.412669 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.412669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.412678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.412732 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.413121 (+ 389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.413122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.413122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.413126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.413137 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.413141 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.413195 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.413261 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.413274 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:08:51.414723 5158 raft_consensus.cc:1261] T 9eba2b49934b495c903a8856818bc6e2 P 8c9657286ba54fa0ae49fb432433cc34 [term 1 FOLLOWER]: Refusing update from remote peer ad546fd766a1443ebb1e6330a821a6b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:51.414723 5420 raft_consensus.cc:1261] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 1 FOLLOWER]: Refusing update from remote peer ad546fd766a1443ebb1e6330a821a6b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:51.415027 5488 consensus_queue.cc:1035] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51e337b9691b4ec2840c47cfd9030d11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:51.415169 5490 consensus_queue.cc:1035] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9657286ba54fa0ae49fb432433cc34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:51.417174 5493 mvcc.cc:205] Tried to move back new op lower bound from 7332738381472509952 to 7332738381418672128. Current Snapshot: MvccSnapshot[applied={T|T < 7332738381472509952}]
I20260924 03:08:51.419166 5016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:51.419245 5016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:51.419888 5016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c975b93344e48b6836c6b4fc38a64a6: Sending DeleteTablet for 3 replicas of tablet 9eba2b49934b495c903a8856818bc6e2
I20260924 03:08:51.420270 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eba2b49934b495c903a8856818bc6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:51 UTC) from {username='slave'} at 127.0.0.1:52574
I20260924 03:08:51.420284 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eba2b49934b495c903a8856818bc6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:51 UTC) from {username='slave'} at 127.0.0.1:42596
I20260924 03:08:51.420450 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eba2b49934b495c903a8856818bc6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:51 UTC) from {username='slave'} at 127.0.0.1:53592
I20260924 03:08:51.420485 5503 tablet_replica.cc:330] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: stopping tablet replica
I20260924 03:08:51.420475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5076
I20260924 03:08:51.420629 5505 tablet_replica.cc:330] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: stopping tablet replica
I20260924 03:08:51.420703 5505 raft_consensus.cc:2229] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:51.420800 5505 raft_consensus.cc:2258] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:51.420614 5503 raft_consensus.cc:2229] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:51.421262 5505 ts_tablet_manager.cc:1905] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:51.421356 5503 raft_consensus.cc:2258] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:51.421710 5503 ts_tablet_manager.cc:1905] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:51.423106 5505 ts_tablet_manager.cc:1918] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:51.423207 5505 log.cc:1196] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/9eba2b49934b495c903a8856818bc6e2
I20260924 03:08:51.423511 5505 ts_tablet_manager.cc:1939] T 9eba2b49934b495c903a8856818bc6e2 P ad546fd766a1443ebb1e6330a821a6b8: Deleting consensus metadata
I20260924 03:08:51.423799 5503 ts_tablet_manager.cc:1918] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:51.423861 5002 catalog_manager.cc:4841] TS ad546fd766a1443ebb1e6330a821a6b8 (127.0.98.2:39405): tablet 9eba2b49934b495c903a8856818bc6e2 (table test-table [id=346810cebaca44b59cd619f2fc30388c]) successfully deleted
I20260924 03:08:51.423959 5503 log.cc:1196] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/9eba2b49934b495c903a8856818bc6e2
I20260924 03:08:51.424284 5503 ts_tablet_manager.cc:1939] T 9eba2b49934b495c903a8856818bc6e2 P 51e337b9691b4ec2840c47cfd9030d11: Deleting consensus metadata
I20260924 03:08:51.424690 5002 catalog_manager.cc:4841] TS 51e337b9691b4ec2840c47cfd9030d11 (127.0.98.3:44481): tablet 9eba2b49934b495c903a8856818bc6e2 (table test-table [id=346810cebaca44b59cd619f2fc30388c]) successfully deleted
W20260924 03:08:51.427211 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35315 (error 108)
I20260924 03:08:51.427685 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.427569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35315 (local address 127.0.0.1:42604)
0924 03:08:51.427619 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.427659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 03:08:51.427754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5207
W20260924 03:08:51.427790 5002 catalog_manager.cc:4584] TS 8c9657286ba54fa0ae49fb432433cc34 (127.0.98.1:35315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9eba2b49934b495c903a8856818bc6e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35315: connect: Connection refused (error 111)
I20260924 03:08:51.432370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5338
I20260924 03:08:51.436818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 4985
2026-09-24T03:08:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (578 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 03:08:51.455421 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:08:51.456346 392 webserver.cc:460] Webserver started at https://127.0.0.1:43537/ using document root <none> and password file <none>
I20260924 03:08:51.456413 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:08:51.456573 392 webserver.cc:460] Webserver started at http://127.0.0.1:45579/ using document root <none> and password file <none>
2026-09-24T03:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:51Z Disabled control of system clock
I20260924 03:08:51.463572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37121
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37121
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43537/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/testchainca.cert with env {}
W20260924 03:08:51.536760 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.536919 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.536937 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.537844 5512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:51.537873 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.537884 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.537894 5512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:51.537904 5512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:08:51.537915 5512 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:08:51.537930 5512 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43537/jwks.json
I20260924 03:08:51.539270 5512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43537/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.539435 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.539570 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.543762 5517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.543762 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.543759 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.544044 5512 server_base.cc:1034] running on GCE node
I20260924 03:08:51.544230 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.544425 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.545583 5512 hybrid_clock.cc:648] HybridClock initialized: now 1790219331545564 us; error 34 us; skew 500 ppm
I20260924 03:08:51.546782 5512 webserver.cc:460] Webserver started at http://127.0.98.62:35525/ using document root <none> and password file <none>
I20260924 03:08:51.546994 5512 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.547065 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.547216 5512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.548101 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "f5396078c938456489242679ce1ed99b"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "84cb25fcb91a694e5126516c3d915b3b"
server_key_iv: "965f8752cf488cac898a62559a80a992"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.548426 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5396078c938456489242679ce1ed99b"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "84cb25fcb91a694e5126516c3d915b3b"
server_key_iv: "965f8752cf488cac898a62559a80a992"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.549614 5512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:08:51.550346 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.550518 5512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:51.550611 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "f5396078c938456489242679ce1ed99b"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "84cb25fcb91a694e5126516c3d915b3b"
server_key_iv: "965f8752cf488cac898a62559a80a992"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.550690 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.558043 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.558326 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.558465 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.562039 5512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37121
I20260924 03:08:51.562084 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37121 every 8 connection(s)
I20260924 03:08:51.562377 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:51.563171 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.564918 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b: Bootstrap starting.
I20260924 03:08:51.565351 5579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.565562 5579 log.cc:824] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.566123 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b: No bootstrap required, opened a new log
I20260924 03:08:51.567034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5512
I20260924 03:08:51.567117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:51.567286 392 external_mini_cluster.cc:1428] Setting key aee10fd6933043647b0c7b4617bb7111
I20260924 03:08:51.567988 5579 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } }
I20260924 03:08:51.568096 5579 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.568138 5579 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5396078c938456489242679ce1ed99b, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.568270 5579 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } }
I20260924 03:08:51.568346 5579 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:51.568379 5579 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:51.568424 5579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.568850 5579 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } }
I20260924 03:08:51.568982 5579 leader_election.cc:302] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5396078c938456489242679ce1ed99b; no voters:
I20260924 03:08:51.569142 5579 leader_election.cc:288] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:51.569191 5584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:51.569320 5584 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [term 1 LEADER]: Becoming Leader. State: Replica: f5396078c938456489242679ce1ed99b, State: Running, Role: LEADER
I20260924 03:08:51.569375 5579 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:51.569406 5584 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } }
I20260924 03:08:51.569682 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5396078c938456489242679ce1ed99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } } }
I20260924 03:08:51.569752 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.569674 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5396078c938456489242679ce1ed99b. Latest consensus state: current_term: 1 leader_uuid: "f5396078c938456489242679ce1ed99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5396078c938456489242679ce1ed99b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37121 } } }
I20260924 03:08:51.569923 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.570169 5590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:51.570679 5590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:51.572355 5590 catalog_manager.cc:1347] Generated new cluster ID: 08e7b73a1a7f4680a6edb613e432cc76
I20260924 03:08:51.572413 5590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:51.572989 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.567700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.98.62:37121)
0924 03:08:51.567884 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.567890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.567916 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:08:51.567992 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.567997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.568038 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.568167 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.568385 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.569938 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.571062 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.571541 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.571673 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.572323 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.572327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.572327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.572527 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.572566 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.572579 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.572647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.572751 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.572825 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260924 03:08:51.573650 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:08:51.589366 5590 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:51.589844 5590 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:51.593032 5590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5396078c938456489242679ce1ed99b: Generated new TSK 0
I20260924 03:08:51.593181 5590 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:08:51.615929 5609 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 03:08:51.616030 5609 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:08:51.616192 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.602348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.98.62:37121)
0924 03:08:51.602406 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.602407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.602521 (+ 114us) server_negotiation.cc:407] Connection header received
0924 03:08:51.602579 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.602580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.602600 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.602672 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:08:51.602810 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.603589 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.604309 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.604559 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.604619 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.604765 (+ 146us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:08:51.616022 (+ 11257us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:08:51.616135 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 03:08:51.616240 5583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55210: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 03:08:51.616605 5608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:51.602278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37121 (local address 127.0.0.1:55210)
0924 03:08:51.602457 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.602466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.602519 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.602687 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.602688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.602689 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:08:51.602748 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:51.602749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.603666 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.603667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:51.604241 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:51.604242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.604625 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.604628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:51.604666 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.604680 (+ 14us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:08:51.616177 (+ 11497us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:08:51.616351 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:37121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43537/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 03:08:51.617172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5512
2026-09-24T03:08:51Z chronyd exiting
I20260924 03:08:51.652681 392 test_util.cc:175] -----------------------------------------------
I20260924 03:08:51.652740 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219310260848-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (198 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 03:08:51.653409 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:08:51.653986 392 webserver.cc:460] Webserver started at https://127.0.0.1:41967/ using document root <none> and password file <none>
I20260924 03:08:51.654028 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:08:51.654237 392 webserver.cc:460] Webserver started at http://127.0.0.1:36067/ using document root <none> and password file <none>
2026-09-24T03:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:51Z Disabled control of system clock
I20260924 03:08:51.661036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35721
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/testchainca.cert with env {}
W20260924 03:08:51.736917 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.737138 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.737171 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.738322 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:51.738374 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.738395 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.738422 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:51.738445 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:08:51.738472 5615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:08:51.738494 5615 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41967/jwks.json
I20260924 03:08:51.739903 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.740155 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.740351 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.744911 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.744889 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.744937 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.745209 5615 server_base.cc:1034] running on GCE node
I20260924 03:08:51.745414 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.745651 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.746778 5615 hybrid_clock.cc:648] HybridClock initialized: now 1790219331746760 us; error 33 us; skew 500 ppm
I20260924 03:08:51.747947 5615 webserver.cc:460] Webserver started at http://127.0.98.62:40883/ using document root <none> and password file <none>
I20260924 03:08:51.748131 5615 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.748168 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.748250 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.749140 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "7962934ea08b4f5cabde7e07f24b1bad"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "5d5f358f430d29d694d85cf50be45ad8"
server_key_iv: "43e4226ccd5cc0a39b45870fee61ab3b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.749414 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "7962934ea08b4f5cabde7e07f24b1bad"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "5d5f358f430d29d694d85cf50be45ad8"
server_key_iv: "43e4226ccd5cc0a39b45870fee61ab3b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.750716 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:51.751368 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.751531 5615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:51.751605 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "7962934ea08b4f5cabde7e07f24b1bad"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "5d5f358f430d29d694d85cf50be45ad8"
server_key_iv: "43e4226ccd5cc0a39b45870fee61ab3b"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.751654 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.760430 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.760632 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.760721 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.764108 5615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35721
I20260924 03:08:51.764161 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35721 every 8 connection(s)
I20260924 03:08:51.764456 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:51.764987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5615
I20260924 03:08:51.765085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:51.765268 392 external_mini_cluster.cc:1428] Setting key 77751fa5692703fcbef276df21ce70f2
I20260924 03:08:51.765522 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.768690 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad: Bootstrap starting.
I20260924 03:08:51.769347 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.769635 5682 log.cc:824] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.770350 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad: No bootstrap required, opened a new log
I20260924 03:08:51.771070 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.765697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.98.62:35721)
0924 03:08:51.765878 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.765886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.765914 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:08:51.766424 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.766433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.766506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.766647 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.766794 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.768117 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.768844 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.769425 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.769559 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.770429 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.770433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.770433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.770621 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.770656 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.770669 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.770731 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.770798 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.770869 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 03:08:51.772032 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:08:51.772624 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } }
I20260924 03:08:51.772773 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.772819 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7962934ea08b4f5cabde7e07f24b1bad, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.772933 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } }
I20260924 03:08:51.773010 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:51.773039 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:51.773082 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.773573 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } }
I20260924 03:08:51.773655 5682 leader_election.cc:302] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7962934ea08b4f5cabde7e07f24b1bad; no voters:
I20260924 03:08:51.773763 5682 leader_election.cc:288] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:51.773834 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:51.773967 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [term 1 LEADER]: Becoming Leader. State: Replica: 7962934ea08b4f5cabde7e07f24b1bad, State: Running, Role: LEADER
I20260924 03:08:51.774031 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:51.774052 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } }
I20260924 03:08:51.774351 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7962934ea08b4f5cabde7e07f24b1bad. Latest consensus state: current_term: 1 leader_uuid: "7962934ea08b4f5cabde7e07f24b1bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } } }
I20260924 03:08:51.774436 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.774360 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7962934ea08b4f5cabde7e07f24b1bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7962934ea08b4f5cabde7e07f24b1bad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35721 } } }
I20260924 03:08:51.774716 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:51.775422 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:51.776043 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.776698 5693 catalog_manager.cc:1347] Generated new cluster ID: 5653a94ed7c041edb9fbd80344d0d279
I20260924 03:08:51.776743 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:51.798306 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:51.798990 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:51.803727 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7962934ea08b4f5cabde7e07f24b1bad: Generated new TSK 0
I20260924 03:08:51.803879 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:08:51.830341 5712 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 03:08:51.830405 5712 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:08:51.830690 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.817029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.0.98.62:35721)
0924 03:08:51.817091 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.817093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.817211 (+ 118us) server_negotiation.cc:407] Connection header received
0924 03:08:51.817304 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.817305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.817317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.817385 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:08:51.817492 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.818259 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.818995 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.819304 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.819371 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.819496 (+ 125us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:08:51.830498 (+ 11002us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:08:51.830631 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 03:08:51.830744 5685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 03:08:51.830721 5711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:51.816968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35721 (local address 127.0.0.1:40606)
0924 03:08:51.817141 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:51.817153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:51.817226 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:51.817392 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:51.817392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:51.817394 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:08:51.817439 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:51.817440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.818346 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.818347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:51.818912 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:51.818914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.819373 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.819376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:51.819410 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.819414 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:08:51.830587 (+ 11173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:08:51.830665 (+ 78us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260924 03:08:51.830971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5615
2026-09-24T03:08:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 03:08:51.867713 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:08:51.868294 392 webserver.cc:460] Webserver started at https://127.0.0.1:37645/ using document root <none> and password file <none>
I20260924 03:08:51.868330 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:08:51.868453 392 webserver.cc:460] Webserver started at http://127.0.0.1:41683/ using document root <none> and password file <none>
2026-09-24T03:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:51Z Disabled control of system clock
I20260924 03:08:51.875028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44129
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:08:51.951205 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:51.951403 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:51.951437 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:51.952507 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:51.952566 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:51.952589 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:51.952620 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:51.952643 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:08:51.952672 5718 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:08:51.952713 5718 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37645/jwks.json
I20260924 03:08:51.953940 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:51.954211 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:51.954466 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:51.958591 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.958689 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:51.958600 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:51.958820 5718 server_base.cc:1034] running on GCE node
I20260924 03:08:51.959084 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:51.959352 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:51.960517 5718 hybrid_clock.cc:648] HybridClock initialized: now 1790219331960500 us; error 31 us; skew 500 ppm
I20260924 03:08:51.961650 5718 webserver.cc:460] Webserver started at http://127.0.98.62:42749/ using document root <none> and password file <none>
I20260924 03:08:51.961881 5718 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:51.961966 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:51.962095 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:51.963105 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "bcc9a203202d4e7bae764e4daed8e489"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "0ba53f739310dde968079eec6c9f2981"
server_key_iv: "bcb5d9649e4eda8e0adea3f51f221a5c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.963469 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcc9a203202d4e7bae764e4daed8e489"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "0ba53f739310dde968079eec6c9f2981"
server_key_iv: "bcb5d9649e4eda8e0adea3f51f221a5c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.964669 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:08:51.965308 5732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:51.965478 5718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:51.965590 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "bcc9a203202d4e7bae764e4daed8e489"
format_stamp: "Formatted at 2026-09-24 03:08:51 on dist-test-slave-jtnz"
server_key: "0ba53f739310dde968079eec6c9f2981"
server_key_iv: "bcb5d9649e4eda8e0adea3f51f221a5c"
server_key_version: "encryptionkey@0"
I20260924 03:08:51.965672 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:51.980752 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:51.981017 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:51.981149 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:51.984748 5718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44129
I20260924 03:08:51.984798 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44129 every 8 connection(s)
I20260924 03:08:51.985133 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:51.985749 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:51.988056 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489: Bootstrap starting.
I20260924 03:08:51.988571 5785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:51.988884 5785 log.cc:824] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:51.988965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5718
I20260924 03:08:51.989058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:51.989230 392 external_mini_cluster.cc:1428] Setting key 218f1559b93af7c3422db4c646b503ab
I20260924 03:08:51.989639 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489: No bootstrap required, opened a new log
I20260924 03:08:51.991317 5785 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } }
I20260924 03:08:51.991456 5785 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:51.991518 5785 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc9a203202d4e7bae764e4daed8e489, State: Initialized, Role: FOLLOWER
I20260924 03:08:51.991647 5785 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } }
I20260924 03:08:51.991734 5785 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:51.991773 5785 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:51.991818 5785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:51.992440 5785 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } }
I20260924 03:08:51.992548 5785 leader_election.cc:302] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc9a203202d4e7bae764e4daed8e489; no voters:
I20260924 03:08:51.992702 5785 leader_election.cc:288] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:51.992738 5790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:51.992884 5790 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [term 1 LEADER]: Becoming Leader. State: Replica: bcc9a203202d4e7bae764e4daed8e489, State: Running, Role: LEADER
I20260924 03:08:51.992990 5785 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:51.992997 5790 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } }
I20260924 03:08:51.993453 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc9a203202d4e7bae764e4daed8e489. Latest consensus state: current_term: 1 leader_uuid: "bcc9a203202d4e7bae764e4daed8e489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } } }
I20260924 03:08:51.993575 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.993754 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc9a203202d4e7bae764e4daed8e489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc9a203202d4e7bae764e4daed8e489" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44129 } } }
I20260924 03:08:51.993866 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:51.993952 5799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:51.994452 5799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:51.996105 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:51.989719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.0.98.62:44129)
0924 03:08:51.989903 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:51.989910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:51.989932 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:08:51.991305 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:51.991310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:51.991360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:51.991458 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:51.991719 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.992671 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.993753 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:51.994408 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:51.994501 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:51.995413 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:51.995416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:51.995416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:51.995612 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:51.995650 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:51.995664 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:51.995727 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:51.995842 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 03:08:51.995923 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260924 03:08:51.996340 5799 catalog_manager.cc:1347] Generated new cluster ID: a994114b44e942de9e6d504a3615e673
I20260924 03:08:51.996387 5799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:51.996951 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:08:52.007184 5799 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:52.007689 5799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:52.013636 5799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcc9a203202d4e7bae764e4daed8e489: Generated new TSK 0
I20260924 03:08:52.013782 5799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:52.021662 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:52.018159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.0.98.62:44129)
0924 03:08:52.018216 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:52.018217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:52.018296 (+ 79us) server_negotiation.cc:407] Connection header received
0924 03:08:52.018366 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:52.018366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:52.018387 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:52.018436 (+ 49us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:08:52.018586 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.019041 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.020180 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.020349 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.020400 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:52.020591 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:08:52.021487 (+ 896us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:08:52.021604 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 03:08:52.021705 5789 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 03:08:52.021780 5814 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:52.018077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44129 (local address 127.0.0.1:53274)
0924 03:08:52.018236 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:52.018246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:52.018306 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:52.018452 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:52.018452 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:52.018454 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:08:52.018519 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:52.018521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.019120 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.019121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:52.020097 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:52.020099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.020425 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.020429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:52.020491 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:52.020505 (+ 14us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:08:52.021586 (+ 1081us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:08:52.021678 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44129: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260924 03:08:52.022076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5718
E20260924 03:08:52.033856 5815 webserver.cc:533] Webserver: sslize failed:
E20260924 03:08:52.033905 5815 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T03:08:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (192 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T03:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:52Z Disabled control of system clock
I20260924 03:08:52.066692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34869
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34869
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/testchainca.cert with env {}
W20260924 03:08:52.138161 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:52.138391 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:52.138423 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:52.139577 5819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:52.139647 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:52.139668 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:52.139699 5819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:52.139731 5819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:08:52.139756 5819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:08:52.139786 5819 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 03:08:52.141178 5819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:52.141422 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:52.141621 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:52.145875 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:52.145855 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:52.145876 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:52.146138 5819 server_base.cc:1034] running on GCE node
I20260924 03:08:52.146500 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:52.146747 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:52.147897 5819 hybrid_clock.cc:648] HybridClock initialized: now 1790219332147889 us; error 35 us; skew 500 ppm
I20260924 03:08:52.149014 5819 webserver.cc:460] Webserver started at http://127.0.98.62:43529/ using document root <none> and password file <none>
I20260924 03:08:52.149266 5819 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:52.149334 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:52.149459 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:52.150499 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "c1bee329c10b43bd89c5b097c7a2cc22"
format_stamp: "Formatted at 2026-09-24 03:08:52 on dist-test-slave-jtnz"
server_key: "76e28b891dab5a931c826bc17ab16b09"
server_key_iv: "21b40288975496a581bbb7cad3f197a1"
server_key_version: "encryptionkey@0"
I20260924 03:08:52.150827 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1bee329c10b43bd89c5b097c7a2cc22"
format_stamp: "Formatted at 2026-09-24 03:08:52 on dist-test-slave-jtnz"
server_key: "76e28b891dab5a931c826bc17ab16b09"
server_key_iv: "21b40288975496a581bbb7cad3f197a1"
server_key_version: "encryptionkey@0"
I20260924 03:08:52.151927 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:52.152529 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:52.152699 5819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:52.152791 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "c1bee329c10b43bd89c5b097c7a2cc22"
format_stamp: "Formatted at 2026-09-24 03:08:52 on dist-test-slave-jtnz"
server_key: "76e28b891dab5a931c826bc17ab16b09"
server_key_iv: "21b40288975496a581bbb7cad3f197a1"
server_key_version: "encryptionkey@0"
I20260924 03:08:52.152864 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:52.168798 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:52.169086 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:52.169226 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:52.172793 5819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34869
I20260924 03:08:52.172822 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34869 every 8 connection(s)
I20260924 03:08:52.173130 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:52.174024 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:52.176127 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22: Bootstrap starting.
I20260924 03:08:52.176658 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:52.176972 5886 log.cc:824] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:52.177588 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22: No bootstrap required, opened a new log
I20260924 03:08:52.179185 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } }
I20260924 03:08:52.179311 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:52.179358 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bee329c10b43bd89c5b097c7a2cc22, State: Initialized, Role: FOLLOWER
I20260924 03:08:52.179544 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } }
I20260924 03:08:52.179622 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:52.179672 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:52.179721 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:52.180260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5819
I20260924 03:08:52.180280 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } }
I20260924 03:08:52.180361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:52.180374 5886 leader_election.cc:302] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1bee329c10b43bd89c5b097c7a2cc22; no voters:
I20260924 03:08:52.180531 5886 leader_election.cc:288] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:52.180572 392 external_mini_cluster.cc:1428] Setting key 5cc8a1a3378170b936a841eb509b4123
I20260924 03:08:52.180588 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:52.180719 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [term 1 LEADER]: Becoming Leader. State: Replica: c1bee329c10b43bd89c5b097c7a2cc22, State: Running, Role: LEADER
I20260924 03:08:52.180792 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } }
I20260924 03:08:52.180761 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:52.181082 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1bee329c10b43bd89c5b097c7a2cc22. Latest consensus state: current_term: 1 leader_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } } }
I20260924 03:08:52.181144 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:52.181238 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bee329c10b43bd89c5b097c7a2cc22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34869 } } }
I20260924 03:08:52.181363 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:52.181483 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:52.181990 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:52.183611 5896 catalog_manager.cc:1347] Generated new cluster ID: 272081027fbf4b48a9b0569df6db2952
I20260924 03:08:52.183661 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:52.186866 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:52.181105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.0.98.62:34869)
0924 03:08:52.181356 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:52.181366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:52.182399 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 03:08:52.182668 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:52.182676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:52.182744 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:52.182879 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:52.183261 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.184225 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.184982 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.185402 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.185547 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:52.186215 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:52.186218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:52.186218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:08:52.186390 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:52.186419 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:08:52.186428 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:52.186512 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:52.186600 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 03:08:52.186684 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":63,"threads_started":1}
I20260924 03:08:52.187585 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:08:52.204172 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:52.204749 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:52.212908 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1bee329c10b43bd89c5b097c7a2cc22: Generated new TSK 0
I20260924 03:08:52.213052 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:52.276407 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:52.217664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.0.98.62:34869)
0924 03:08:52.217703 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:52.217705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:52.217790 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:08:52.217841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:52.217842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:52.217852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:52.217925 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:08:52.218030 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.218795 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.219509 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.219837 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.219904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:52.220036 (+ 132us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:08:52.276146 (+ 56110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:08:52.276332 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 03:08:52.276475 5894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 03:08:52.276458 5915 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:52.217593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34869 (local address 127.0.0.1:52974)
0924 03:08:52.217722 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:52.217732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:52.217790 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:52.217928 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:52.217929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:52.217930 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:08:52.217969 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:52.217970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.218866 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.218867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:52.219434 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:52.219435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:52.219912 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:52.219915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:52.219950 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:52.219954 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:08:52.276273 (+ 56319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:08:52.276386 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260924 03:08:52.276799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5819
2026-09-24T03:08:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (233 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (81 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (79 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:52 dist-test-slave-jtnz krb5kdc[5927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:52 dist-test-slave-jtnz krb5kdc[5927](info): set up 2 sockets
Sep 24 03:08:52 dist-test-slave-jtnz krb5kdc[5927](info): commencing operation
krb5kdc: starting...
W20260924 03:08:53.406144 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:08:53.512430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:53.586104 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:53.586279 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:53.586297 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:53.587276 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:53.587306 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:53.587317 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:53.587329 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:53.587340 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:53.588600 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:53.588754 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:53.588914 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:53.593369 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.593341 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.593333 5948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.593600 5940 server_base.cc:1034] running on GCE node
I20260924 03:08:53.593834 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:53.594106 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:53.595263 5940 hybrid_clock.cc:648] HybridClock initialized: now 1790219333595250 us; error 43 us; skew 500 ppm
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:53.601771 5940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:08:53.602233 5940 webserver.cc:460] Webserver started at http://127.0.98.62:41125/ using document root <none> and password file <none>
I20260924 03:08:53.602442 5940 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:53.602512 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:53.602660 5940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:53.603981 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "a3220fe4465e423c8dd271ab64f475bc"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "54dd2d621f4c73a6fb90429602989162"
server_key_iv: "32429189b6fdad483847aa8f4e54ae71"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.604363 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3220fe4465e423c8dd271ab64f475bc"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "54dd2d621f4c73a6fb90429602989162"
server_key_iv: "32429189b6fdad483847aa8f4e54ae71"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.605501 5940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:53.606119 5955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.606269 5940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:53.606356 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "a3220fe4465e423c8dd271ab64f475bc"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "54dd2d621f4c73a6fb90429602989162"
server_key_iv: "32429189b6fdad483847aa8f4e54ae71"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.606422 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:53.616509 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:53.617959 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:53.618100 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:53.621486 5940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32993
I20260924 03:08:53.621522 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32993 every 8 connection(s)
I20260924 03:08:53.621982 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:53.622587 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:53.624447 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: Bootstrap starting.
I20260924 03:08:53.624908 6008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:53.625119 6008 log.cc:824] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:53.625648 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: No bootstrap required, opened a new log
I20260924 03:08:53.626281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 5940
I20260924 03:08:53.626371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:53.626544 392 external_mini_cluster.cc:1428] Setting key 7ef707483566598cd1ba68bc28b2bb48
I20260924 03:08:53.627650 6008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260924 03:08:53.627794 6008 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:53.627825 6008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3220fe4465e423c8dd271ab64f475bc, State: Initialized, Role: FOLLOWER
I20260924 03:08:53.627991 6008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260924 03:08:53.628093 6008 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:53.628177 6008 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:53.628229 6008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:53.628831 6008 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260924 03:08:53.628927 6008 leader_election.cc:302] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3220fe4465e423c8dd271ab64f475bc; no voters:
I20260924 03:08:53.629051 6008 leader_election.cc:288] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:53.629168 6013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:53.629278 6008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:53.629328 6013 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [term 1 LEADER]: Becoming Leader. State: Replica: a3220fe4465e423c8dd271ab64f475bc, State: Running, Role: LEADER
I20260924 03:08:53.629438 6013 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260924 03:08:53.629863 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3220fe4465e423c8dd271ab64f475bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } }
I20260924 03:08:53.629935 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:53.630055 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3220fe4465e423c8dd271ab64f475bc. Latest consensus state: current_term: 1 leader_uuid: "a3220fe4465e423c8dd271ab64f475bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3220fe4465e423c8dd271ab64f475bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } }
I20260924 03:08:53.630155 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc [sys.catalog]: This master's current role is: LEADER
W20260924 03:08:53.630913 6029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:53.630963 6029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:53.631003 6022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:53.631166 6022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:53.632587 6022 catalog_manager.cc:1347] Generated new cluster ID: 222af23007474ec6b2f29cc494133a03
I20260924 03:08:53.632638 6022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:53.637041 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.627064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.62:32993)
0924 03:08:53.627251 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:53.627260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:53.627302 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:08:53.627619 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:53.627626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:53.627679 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:53.627810 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:53.628537 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.629903 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.631065 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.631492 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.631608 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.633927 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:53.633931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:53.633931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:53.633948 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:53.635544 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.635923 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.635925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.635925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.636026 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.636229 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.636229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.636230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.636435 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:53.636621 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:53.636713 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:08:53.636846 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260924 03:08:53.673054 6022 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:53.673535 6022 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:53.680162 6022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3220fe4465e423c8dd271ab64f475bc: Generated new TSK 0
I20260924 03:08:53.680331 6022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:08:53.711040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:53.808741 6034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:53.808916 6034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:53.808938 6034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:53.810056 6034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:53.810112 6034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:53.810138 6034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:53.811517 6034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:53.811668 6034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:53.811803 6034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:53.816143 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.816135 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.816367 6034 server_base.cc:1034] running on GCE node
W20260924 03:08:53.816432 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.816661 6034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:53.816833 6034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:53.817960 6034 hybrid_clock.cc:648] HybridClock initialized: now 1790219333817947 us; error 28 us; skew 500 ppm
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:53.820286 6034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:08:53.820617 6034 webserver.cc:460] Webserver started at http://127.0.98.1:45633/ using document root <none> and password file <none>
I20260924 03:08:53.820788 6034 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:53.820822 6034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:53.820899 6034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:53.821709 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a278facbfac4fa8be0c55a3865a7413"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "32b9f57837848faf7548808d3abedd6d"
server_key_iv: "3a05a099c5042dd0015ade4149e53745"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.822012 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a278facbfac4fa8be0c55a3865a7413"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "32b9f57837848faf7548808d3abedd6d"
server_key_iv: "3a05a099c5042dd0015ade4149e53745"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.823203 6034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:08:53.823791 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.823938 6034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:53.824028 6034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "8a278facbfac4fa8be0c55a3865a7413"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "32b9f57837848faf7548808d3abedd6d"
server_key_iv: "3a05a099c5042dd0015ade4149e53745"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.824100 6034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:53.832624 6034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:53.834640 6034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:53.834780 6034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:53.834969 6034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:53.835215 6034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:53.835263 6034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.835297 6034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:53.835321 6034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.840632 6034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41533
I20260924 03:08:53.840703 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41533 every 8 connection(s)
I20260924 03:08:53.840983 6034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:53.845733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6034
I20260924 03:08:53.845820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:53.845992 392 external_mini_cluster.cc:1428] Setting key 1893df521daea5855f62aaa71094f747
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:53.849649 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.841606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50921 (local address 127.0.98.62:32993)
0924 03:08:53.841663 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:53.841665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:53.841857 (+ 192us) server_negotiation.cc:407] Connection header received
0924 03:08:53.842775 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:53.842775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:53.842787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:53.842838 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:53.843339 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.843814 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.844676 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.844866 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.844928 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.847068 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:53.847070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:53.847070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:53.847078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:53.848340 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.848743 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.848744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.848745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.848803 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.849042 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.849043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.849044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.849151 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:53.849218 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:53.849471 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:08:53.849562 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:08:53.849785 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.841581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.98.1:50921)
0924 03:08:53.841717 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:53.841751 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:53.842692 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:53.842883 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:53.842885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:53.843113 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:53.843274 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.843277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.843886 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.843886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.844609 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.844611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.844947 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.844948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.845043 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.845460 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:53.845468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:53.846942 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:53.848433 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.848440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.848442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.848669 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.848874 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.848875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.848875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.848973 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.849222 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:53.849228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:53.849351 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 03:08:53.849451 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:08:53.849586 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 03:08:53.850234 6163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32993
I20260924 03:08:53.850332 6163 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:53.850543 6163 heartbeater.cc:511] Master 127.0.98.62:32993 requested a full tablet report, sending...
I20260924 03:08:53.851009 5972 ts_manager.cc:194] Registered new tserver with Master: 8a278facbfac4fa8be0c55a3865a7413 (127.0.98.1:41533)
I20260924 03:08:53.851563 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50921
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:08:53.868062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:53.942238 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:53.942436 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:53.942459 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:53.943557 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:53.943593 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:53.943639 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:53.944901 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:53.945096 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:53.945276 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:53.949755 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.949774 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:53.949755 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:53.950047 6168 server_base.cc:1034] running on GCE node
I20260924 03:08:53.950300 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:53.950536 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:53.951671 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790219333951659 us; error 27 us; skew 500 ppm
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:53.953969 6168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:08:53.954424 6168 webserver.cc:460] Webserver started at http://127.0.98.2:39979/ using document root <none> and password file <none>
I20260924 03:08:53.954638 6168 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:53.954679 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:53.954820 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:53.955825 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "d93602c4ae2a499b8178cdfe47645f12"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "9e5d061c2a510afcc56e64d1d98ac22b"
server_key_iv: "e47a8bd5dc960baf9d943172646f2670"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.956172 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d93602c4ae2a499b8178cdfe47645f12"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "9e5d061c2a510afcc56e64d1d98ac22b"
server_key_iv: "e47a8bd5dc960baf9d943172646f2670"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.957441 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:53.958127 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.958343 6168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:53.958432 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "d93602c4ae2a499b8178cdfe47645f12"
format_stamp: "Formatted at 2026-09-24 03:08:53 on dist-test-slave-jtnz"
server_key: "9e5d061c2a510afcc56e64d1d98ac22b"
server_key_iv: "e47a8bd5dc960baf9d943172646f2670"
server_key_version: "encryptionkey@0"
I20260924 03:08:53.958510 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:53.977205 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:53.978991 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:53.979143 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:53.979354 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:53.979652 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:53.979684 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.979707 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:53.979751 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:53.985178 6168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36095
I20260924 03:08:53.985256 6296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36095 every 8 connection(s)
I20260924 03:08:53.985548 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:08:53 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:53.991990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6168
I20260924 03:08:53.992067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:53.992281 392 external_mini_cluster.cc:1428] Setting key b4772c36007b20d6ef444efbf3a0e801
I20260924 03:08:53.994993 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.986279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35129 (local address 127.0.98.62:32993)
0924 03:08:53.986362 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:53.986365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:53.986508 (+ 143us) server_negotiation.cc:407] Connection header received
0924 03:08:53.987459 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:53.987460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:53.987472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:53.987545 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:53.988144 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.988637 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.989618 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.989827 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.989912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.991885 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:53.991886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:53.991887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:53.991894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:53.993628 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.993986 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.993987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.993987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.994050 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:53.994331 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:53.994331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:53.994332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:53.994477 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:53.994549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:53.994864 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:08:53.994925 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:08:53.995239 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:53.986224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.98.2:35129)
0924 03:08:53.986390 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:53.986424 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:53.987349 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:53.987621 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:53.987624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:53.987854 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:53.988047 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.988052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.988734 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.988735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.989529 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:53.989531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:53.989920 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:53.989921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:53.990046 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:53.990506 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:53.990516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:53.991790 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:53.993706 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.993711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.993712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.993907 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.994114 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:53.994114 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:53.994115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:53.994270 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:53.994550 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:53.994556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:53.994714 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 03:08:53.994833 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:08:53.994992 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 03:08:53.995667 6297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32993
I20260924 03:08:53.995756 6297 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:53.995939 6297 heartbeater.cc:511] Master 127.0.98.62:32993 requested a full tablet report, sending...
I20260924 03:08:53.996292 5972 ts_manager.cc:194] Registered new tserver with Master: d93602c4ae2a499b8178cdfe47645f12 (127.0.98.2:36095)
I20260924 03:08:53.996726 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35129
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:08:54.017005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:54.095400 6302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:54.095587 6302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:54.095607 6302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:54.096642 6302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:54.096673 6302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:54.096697 6302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:54.098023 6302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:54.098232 6302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:54.098410 6302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:54.102962 6308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.102974 6310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:54.102974 6307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:54.103205 6302 server_base.cc:1034] running on GCE node
I20260924 03:08:54.103392 6302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:54.103626 6302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:54.104786 6302 hybrid_clock.cc:648] HybridClock initialized: now 1790219334104773 us; error 24 us; skew 500 ppm
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.107168 6302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:08:54.107571 6302 webserver.cc:460] Webserver started at http://127.0.98.3:36079/ using document root <none> and password file <none>
I20260924 03:08:54.107779 6302 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:54.107841 6302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:54.107981 6302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:54.108973 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e13cbd8eaad4936b9e81df14d9102ee"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-jtnz"
server_key: "73d52e5ffaab41f035b9b0336c74aa5f"
server_key_iv: "81216e4b2f18603510dae4990cb3db16"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.109329 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e13cbd8eaad4936b9e81df14d9102ee"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-jtnz"
server_key: "73d52e5ffaab41f035b9b0336c74aa5f"
server_key_iv: "81216e4b2f18603510dae4990cb3db16"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.110628 6302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:54.111302 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.111459 6302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:54.111552 6302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "2e13cbd8eaad4936b9e81df14d9102ee"
format_stamp: "Formatted at 2026-09-24 03:08:54 on dist-test-slave-jtnz"
server_key: "73d52e5ffaab41f035b9b0336c74aa5f"
server_key_iv: "81216e4b2f18603510dae4990cb3db16"
server_key_version: "encryptionkey@0"
I20260924 03:08:54.111625 6302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:54.125463 6302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:54.127676 6302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:54.127851 6302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:54.128049 6302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:54.128314 6302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:54.128367 6302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.128402 6302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:54.128429 6302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:54.134440 6302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34091
I20260924 03:08:54.134470 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34091 every 8 connection(s)
I20260924 03:08:54.134738 6302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:08:54.140817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6302
I20260924 03:08:54.140877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:54.141054 392 external_mini_cluster.cc:1428] Setting key 59ff0475d0816bda1f939a19465e8075
I20260924 03:08:54.143255 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.135396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52265 (local address 127.0.98.62:32993)
0924 03:08:54.135452 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.135454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.135709 (+ 255us) server_negotiation.cc:407] Connection header received
0924 03:08:54.136520 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.136521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.136531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.136597 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.137150 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.137605 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.138554 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.138724 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.138774 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.140595 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.140596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.140596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.140602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.142116 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.142506 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.142507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.142515 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.142563 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.142760 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.142760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.142761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.142853 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.142901 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.143140 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.143196 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:08:54.143389 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.135376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.98.3:52265)
0924 03:08:54.135549 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:54.135612 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:54.136437 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:54.136645 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:54.136647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:54.136852 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:54.137068 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.137073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.137674 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.137674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.138488 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:54.138490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.138791 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.138792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:54.138877 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.139263 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:54.139270 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:54.140515 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:54.142210 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.142215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.142217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.142443 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.142620 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:54.142621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:54.142621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:54.142708 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:54.142927 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:54.142930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:54.143027 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 03:08:54.143115 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:08:54.143215 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260924 03:08:54.143814 6431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32993
I20260924 03:08:54.143919 6431 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:54.144071 6431 heartbeater.cc:511] Master 127.0.98.62:32993 requested a full tablet report, sending...
I20260924 03:08:54.144419 5972 ts_manager.cc:194] Registered new tserver with Master: 2e13cbd8eaad4936b9e81df14d9102ee (127.0.98.3:34091)
I20260924 03:08:54.144793 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52265
I20260924 03:08:54.152565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/test-admin.keytab.
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.165992 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 03:08:54.167423 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:54.166759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.1:41533)
0924 03:08:54.166912 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.166915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.166924 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:54.166945 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.166947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.166959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.167020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.167334 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45474: BlockingRecv error: recv got EOF from 127.0.0.1:45474 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260924 03:08:54.167464 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc)
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.168536 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.169400 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:54.176607 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.169863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.0.98.1:41533)
0924 03:08:54.169915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.169918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.170005 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:08:54.170076 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.170077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.170089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.170159 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.170561 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.171243 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.171945 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.172252 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.172322 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.174289 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.174296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.174296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.174303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.175535 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.175853 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.175857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.175857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.175905 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.176081 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.176082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.176083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.176194 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.176316 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.176425 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.176494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 03:08:54.178140 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:54.177697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.98.1:41533)
0924 03:08:54.177728 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.177729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.177769 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:08:54.177817 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.177817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.177824 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.177868 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.178091 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45502: BlockingRecv error: recv got EOF from 127.0.0.1:45502 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:08:54.178273 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.179136 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.179904 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:54.185747 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.180421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.1:41533)
0924 03:08:54.180458 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.180460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.180467 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:54.180540 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.180541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.180549 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.180603 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.180893 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.181318 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.181910 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.182125 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.182170 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.183920 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.183921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.183922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.183926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.184999 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.185269 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.185270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.185270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.185312 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.185444 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.185445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.185445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.185514 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.185570 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.185650 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.185701 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:08:54.186998 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:54.186600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.98.1:41533)
0924 03:08:54.186632 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.186633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.186676 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:08:54.186722 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.186723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.186728 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.186765 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.186949 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45524: BlockingRecv error: recv got EOF from 127.0.0.1:45524 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:08:54.187156 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219310260848-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.187966 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:54.188700 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:08:54.194649 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:54.189156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.0.98.1:41533)
0924 03:08:54.189223 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:54.189225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:54.189232 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:54.189286 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:54.189287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:54.189293 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:54.189337 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:54.189645 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.190049 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.190708 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:54.190857 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:54.190903 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:54.192664 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:54.192665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:54.192665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:54.192670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:54.193687 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.193986 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.193987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.193987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.194032 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:54.194221 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:54.194222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:54.194222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:54.194303 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:54.194367 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:54.194471 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:08:54.194545 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:08:54.195202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6034
I20260924 03:08:54.200373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6168
I20260924 03:08:54.204367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6302
I20260924 03:08:54.208117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 5940
2026-09-24T03:08:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1771 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[6451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[6451](info): set up 2 sockets
Sep 24 03:08:54 dist-test-slave-jtnz krb5kdc[6451](info): commencing operation
krb5kdc: starting...
W20260924 03:08:55.186102 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 03:08:55.290714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:55.364073 6464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:55.364233 6464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:55.364252 6464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:55.365255 6464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:55.365293 6464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:55.365305 6464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:55.365316 6464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:55.365366 6464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:55.366642 6464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:55.366837 6464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:55.367040 6464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:55.371315 6469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.371317 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.371364 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.371416 6464 server_base.cc:1034] running on GCE node
I20260924 03:08:55.371739 6464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:55.371987 6464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:55.373137 6464 hybrid_clock.cc:648] HybridClock initialized: now 1790219335373134 us; error 38 us; skew 500 ppm
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:55.375643 6464 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 03:08:55.376055 6464 webserver.cc:460] Webserver started at http://127.0.98.62:41895/ using document root <none> and password file <none>
I20260924 03:08:55.376272 6464 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:55.376331 6464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:55.376462 6464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:55.377328 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "630475a08f9b3e031cd03629f8054b0e"
server_key_iv: "7cfb915ae7e022638d1e823b7e92ab1a"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.377650 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "630475a08f9b3e031cd03629f8054b0e"
server_key_iv: "7cfb915ae7e022638d1e823b7e92ab1a"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.378911 6464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:08:55.379575 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.379732 6464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:55.379823 6464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "630475a08f9b3e031cd03629f8054b0e"
server_key_iv: "7cfb915ae7e022638d1e823b7e92ab1a"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.379896 6464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:55.404147 6464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:55.406304 6464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:55.406455 6464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:55.410048 6464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37777
I20260924 03:08:55.410113 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37777 every 8 connection(s)
I20260924 03:08:55.410432 6464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:55.411273 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:55.413223 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Bootstrap starting.
I20260924 03:08:55.413692 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:55.413892 6532 log.cc:824] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:55.414505 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: No bootstrap required, opened a new log
I20260924 03:08:55.414578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6464
I20260924 03:08:55.414688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:55.414839 392 external_mini_cluster.cc:1428] Setting key 492e5f8aa5b1142936fa1c03d22f6124
I20260924 03:08:55.416352 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } }
I20260924 03:08:55.416497 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:55.416536 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f19f0db7ee04f8c9ccbc5bf0bf1435c, State: Initialized, Role: FOLLOWER
I20260924 03:08:55.416694 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } }
I20260924 03:08:55.416775 6532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:55.416805 6532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:55.416853 6532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:55.417675 6532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } }
I20260924 03:08:55.417750 6532 leader_election.cc:302] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f19f0db7ee04f8c9ccbc5bf0bf1435c; no voters:
I20260924 03:08:55.417927 6532 leader_election.cc:288] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:55.417943 6537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:55.418035 6537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 LEADER]: Becoming Leader. State: Replica: 9f19f0db7ee04f8c9ccbc5bf0bf1435c, State: Running, Role: LEADER
I20260924 03:08:55.418102 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } }
I20260924 03:08:55.418164 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:55.418399 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } }
I20260924 03:08:55.418571 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:55.418740 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } }
I20260924 03:08:55.418807 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:55.419071 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:55.419600 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:55.421144 6545 catalog_manager.cc:1347] Generated new cluster ID: 3a5f43f785ce478bb23545154c6fd329
I20260924 03:08:55.421536 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:55.425865 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.415407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.98.62:37777)
0924 03:08:55.415574 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.415580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.415601 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:08:55.416341 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.416346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.416387 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.416516 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.417221 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.418225 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.419232 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.419663 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.419757 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.422519 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.422522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.422523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.422533 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.424304 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.424697 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.424700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.424700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.424805 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.425000 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.425000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.425001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.425272 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.425415 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.425576 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.425684 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 03:08:55.435746 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:55.436206 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:55.439203 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Generated new TSK 0
I20260924 03:08:55.439337 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 03:08:55.469220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:55.567864 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:55.568137 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:55.568197 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:55.569269 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:55.569310 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:55.569355 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:55.570856 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:55.571070 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:55.571283 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:55.575966 6558 server_base.cc:1034] running on GCE node
W20260924 03:08:55.575973 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.575953 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.575944 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.576303 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:55.576524 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:55.577672 6558 hybrid_clock.cc:648] HybridClock initialized: now 1790219335577653 us; error 30 us; skew 500 ppm
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:55.579952 6558 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:08:55.580317 6558 webserver.cc:460] Webserver started at http://127.0.98.1:36723/ using document root <none> and password file <none>
I20260924 03:08:55.580480 6558 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:55.580546 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:55.580628 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:55.581521 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d6f6aa781794f29af39e98efe90e101"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "bb1831ec44c40ed34d3d861af743f237"
server_key_iv: "453e708d7d22840b82293cb0c4db999d"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.581796 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d6f6aa781794f29af39e98efe90e101"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "bb1831ec44c40ed34d3d861af743f237"
server_key_iv: "453e708d7d22840b82293cb0c4db999d"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.583235 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:55.583992 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.584141 6558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:55.584220 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "5d6f6aa781794f29af39e98efe90e101"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "bb1831ec44c40ed34d3d861af743f237"
server_key_iv: "453e708d7d22840b82293cb0c4db999d"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.584271 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:55.595379 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:55.597517 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:55.597620 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:55.597782 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:55.598001 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:55.598029 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.598049 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:55.598061 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.603536 6558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46103
I20260924 03:08:55.603590 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46103 every 8 connection(s)
I20260924 03:08:55.603816 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:55.604318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6558
I20260924 03:08:55.604415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:08:55.604614 392 external_mini_cluster.cc:1428] Setting key 91321bc66eee24f96717ac30dd69d81d
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:08:55.612713 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.604723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36895 (local address 127.0.98.62:37777)
0924 03:08:55.604804 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.604807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.604818 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:55.605239 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.605240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.605253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.605302 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.606117 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.606618 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.607657 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.607873 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.607906 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.609955 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.609957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.609957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.609964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.611355 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.611739 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.611741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.611741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.611820 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.612057 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.612058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.612058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.612172 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.612241 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.612525 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.612623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:08:55.612883 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.604476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:36895)
0924 03:08:55.604631 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.604663 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.605147 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.605368 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.605371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.605856 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:55.606024 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.606028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.606719 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.606720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.607513 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.607515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.607982 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.607982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.608086 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.608559 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:55.608567 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:55.609860 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:55.611448 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.611454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.611456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.611665 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.611875 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.611875 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.611876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.611987 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.612244 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:55.612251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:55.612380 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.612498 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.612661 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260924 03:08:55.613302 6687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:55.613454 6687 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:55.613647 6687 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 03:08:55.614066 6496 ts_manager.cc:194] Registered new tserver with Master: 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
I20260924 03:08:55.614663 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 03:08:55.626600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:55.702750 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:55.702941 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:55.702965 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:55.704015 6692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:55.704047 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:55.704071 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:55.705283 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:55.705433 6692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:55.705586 6692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:55.710042 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.710130 6692 server_base.cc:1034] running on GCE node
W20260924 03:08:55.710042 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.710068 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.710471 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:55.710701 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:55.711836 6692 hybrid_clock.cc:648] HybridClock initialized: now 1790219335711826 us; error 27 us; skew 500 ppm
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:55.714016 6692 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:08:55.714455 6692 webserver.cc:460] Webserver started at http://127.0.98.2:39695/ using document root <none> and password file <none>
I20260924 03:08:55.714673 6692 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:55.714715 6692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:55.714845 6692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:55.715762 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "d2114174f5a1065932eee340ff86eff4"
server_key_iv: "37af3ddfa0d68f1a84888090ab4eefb4"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.716104 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "d2114174f5a1065932eee340ff86eff4"
server_key_iv: "37af3ddfa0d68f1a84888090ab4eefb4"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.717294 6692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:55.717911 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.718151 6692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:55.718258 6692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "d2114174f5a1065932eee340ff86eff4"
server_key_iv: "37af3ddfa0d68f1a84888090ab4eefb4"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.718328 6692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:55.729791 6692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:55.731424 6692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:55.731591 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:55.731801 6692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:55.732071 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:55.732103 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.732122 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:55.732168 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.737983 6692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43695
I20260924 03:08:55.738016 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43695 every 8 connection(s)
I20260924 03:08:55.738340 6692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:55.740391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6692
I20260924 03:08:55.740489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:08:55.740661 392 external_mini_cluster.cc:1428] Setting key f83b6b5edf8b2c7318c4c96ad5acc5de
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:55.747242 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.738929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36731 (local address 127.0.98.62:37777)
0924 03:08:55.738969 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.738971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.739157 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:08:55.739596 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.739597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.739607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.739659 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.740478 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.741155 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.742166 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.742383 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.742468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.744610 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.744612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.744612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.744619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.745916 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.746387 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.746388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.746389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.746444 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.746674 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.746675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.746675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.746777 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.746850 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.747085 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.747162 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:55.747404 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.738877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:36731)
0924 03:08:55.739040 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.739076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.739516 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.739729 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.739732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.740221 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:55.740382 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.740385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.741239 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.741240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.742066 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.742068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.742475 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.742476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.742615 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.743062 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:55.743069 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:55.744519 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:55.746007 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.746013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.746015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.746309 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.746511 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.746512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.746512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.746611 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.746842 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:55.746848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:55.746970 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.747066 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.747199 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260924 03:08:55.747831 6821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:55.747928 6821 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:55.748101 6821 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
I20260924 03:08:55.748458 6496 ts_manager.cc:194] Registered new tserver with Master: 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:08:55.748831 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:08:55.763113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:55.835815 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:55.835990 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:55.836014 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:55.837093 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:55.837132 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:55.837157 6826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:55.838503 6826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:55.838691 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:55.838856 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:55.843532 6834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.843559 6832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.843554 6831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.843680 6826 server_base.cc:1034] running on GCE node
I20260924 03:08:55.843887 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:55.844128 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:55.845286 6826 hybrid_clock.cc:648] HybridClock initialized: now 1790219335845276 us; error 29 us; skew 500 ppm
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:55.847540 6826 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:08:55.847909 6826 webserver.cc:460] Webserver started at http://127.0.98.3:40315/ using document root <none> and password file <none>
I20260924 03:08:55.848126 6826 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:55.848191 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:55.848335 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:55.849249 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee73197bf4814bdf8180f1ace4605ccf"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "33983e4c1b63853c9f1d12061cf28c87"
server_key_iv: "431d80925dbaebaaab8fb9548d283876"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.849624 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee73197bf4814bdf8180f1ace4605ccf"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "33983e4c1b63853c9f1d12061cf28c87"
server_key_iv: "431d80925dbaebaaab8fb9548d283876"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.850878 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:55.851548 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.851702 6826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.851785 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "ee73197bf4814bdf8180f1ace4605ccf"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "33983e4c1b63853c9f1d12061cf28c87"
server_key_iv: "431d80925dbaebaaab8fb9548d283876"
server_key_version: "encryptionkey@0"
I20260924 03:08:55.851859 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:55.864159 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:55.865613 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:55.865777 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:55.865979 6826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:55.866324 6826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:55.866375 6826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.866415 6826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:55.866442 6826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:55.871954 6826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46515
I20260924 03:08:55.871989 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46515 every 8 connection(s)
I20260924 03:08:55.872258 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:55.876684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6826
I20260924 03:08:55.876763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:08:55.876936 392 external_mini_cluster.cc:1428] Setting key 19b214663149af16b537382c36d8a6ad
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:55.880120 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.872872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45845 (local address 127.0.98.62:37777)
0924 03:08:55.872914 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:55.872915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:55.873084 (+ 169us) server_negotiation.cc:407] Connection header received
0924 03:08:55.873456 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:55.873458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:55.873467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:55.873520 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:55.874329 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.874802 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.875666 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.875832 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.875877 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.877883 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:55.877885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:55.877885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:55.877893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:55.879039 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.879361 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.879362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.879362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.879409 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:55.879609 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:55.879610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:55.879610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:55.879741 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:55.879793 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:55.879997 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:08:55.880060 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:55.880263 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:55.872855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:45845)
0924 03:08:55.872984 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:55.873012 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:55.873385 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:55.873565 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:55.873568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:55.874056 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:55.874266 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.874270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.874872 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.874873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.875600 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:55.875602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:55.875895 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:55.875896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:55.875980 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:55.876340 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:55.876347 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:55.877772 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:55.879100 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.879105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.879107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.879294 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.879463 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:55.879464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:55.879464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:55.879550 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:55.879796 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:55.879800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:55.879895 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 03:08:55.879978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:08:55.880079 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
I20260924 03:08:55.880651 6955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:55.880748 6955 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:55.880903 6955 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
I20260924 03:08:55.881187 6496 ts_manager.cc:194] Registered new tserver with Master: ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3:46515)
I20260924 03:08:55.881525 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45845
I20260924 03:08:55.888540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 03:08:55.910945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42329
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37777,127.0.98.61:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:55.987000 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:55.987260 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:55.987308 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:55.988428 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:55.988488 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:55.988521 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:55.988554 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:55.988575 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:55.990046 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37777,127.0.98.61:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
W20260924 03:08:55.990180 6960 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37777,127.0.98.61:42329'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 03:08:55.990350 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:55.990553 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:55.994760 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:55.994784 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.994776 6960 server_base.cc:1034] running on GCE node
W20260924 03:08:55.994776 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:55.995107 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:55.995424 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:55.996577 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790219335996561 us; error 27 us; skew 500 ppm
Sep 24 03:08:55 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:55.998811 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 03:08:55.999197 6960 webserver.cc:460] Webserver started at http://127.0.98.61:45891/ using document root <none> and password file <none>
I20260924 03:08:55.999423 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:55.999496 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:55.999639 6960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:56.000586 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/instance:
uuid: "03bd6c71fc1d465e94661252699862e2"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "4827bb8ea7f97af0d0af02ae9bccb4c6"
server_key_iv: "9f5a8fb2190ff16d7af68cf8fabc3638"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.000913 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal/instance:
uuid: "03bd6c71fc1d465e94661252699862e2"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "4827bb8ea7f97af0d0af02ae9bccb4c6"
server_key_iv: "9f5a8fb2190ff16d7af68cf8fabc3638"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.002122 6960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:08:56.002825 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.003008 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:56.003100 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "03bd6c71fc1d465e94661252699862e2"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "4827bb8ea7f97af0d0af02ae9bccb4c6"
server_key_iv: "9f5a8fb2190ff16d7af68cf8fabc3638"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.003180 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.011794 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.013324 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:56.013458 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.016976 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42329
I20260924 03:08:56.017004 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42329 every 8 connection(s)
I20260924 03:08:56.017278 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:08:56.018139 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:56.019533 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:08:56.024569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 6960
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:56.028384 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.020152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.98.62:37777)
0924 03:08:56.020229 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.020231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.020393 (+ 162us) server_negotiation.cc:407] Connection header received
0924 03:08:56.020850 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.020852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.020864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.020938 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.021538 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.022024 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.023053 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.023272 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.023354 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.025425 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.025426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.025426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.025432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.026998 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.027437 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.027438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.027438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.027493 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.027723 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.027724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.027724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.027858 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.027910 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.028191 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.028264 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:08:56.028420 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.020116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58216)
0924 03:08:56.020289 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.020318 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.020745 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.021004 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.021010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.021283 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.021439 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.021443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.022112 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.022113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.022947 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.022949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.023374 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.023375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.023479 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.023897 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.023904 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.025334 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.027086 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.027092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.027094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.027358 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.027553 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.027554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.027554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.027668 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.027926 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.027932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.028075 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.028149 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.028250 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260924 03:08:56.029027 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:08:56.033726 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.025053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.98.61:42329)
0924 03:08:56.025166 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.025170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.025191 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:08:56.025208 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.025209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.025222 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.025282 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.025909 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.026998 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.027973 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.028320 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.028382 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.030906 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.030912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.030912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.030917 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.032450 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.032827 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.032830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.032830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.032882 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.033091 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.033092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.033092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.033219 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.033373 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.033423 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.033494 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260924 03:08:56.035569 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.029244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43398)
0924 03:08:56.029285 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.029294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.029324 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.029597 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.029597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.029728 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.029761 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.029762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.030453 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.030454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.031158 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.031160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.031673 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.031674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.031749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.032115 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.032120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.033254 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.034834 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.034835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.034835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.035007 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.035156 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.035157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.035157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.035201 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.035373 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.035374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.035402 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.035448 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.035516 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:08:56.035681 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.029299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.61:42329)
0924 03:08:56.029480 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.029481 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.029486 (+ 5us) server_negotiation.cc:407] Connection header received
0924 03:08:56.029499 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.029500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.029522 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.029596 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.029835 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.030374 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.031246 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.031602 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.031665 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.033344 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.033345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.033346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.033351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.034770 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.035063 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.035063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.035063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.035105 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.035237 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.035238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.035238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.035326 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.035367 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.035463 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.035556 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260924 03:08:56.036667 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Bootstrap starting.
I20260924 03:08:56.037216 7028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:56.037468 7028 log.cc:824] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:56.038069 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: No bootstrap required, opened a new log
I20260924 03:08:56.039299 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } }
I20260924 03:08:56.039443 7028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:56.039489 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd6c71fc1d465e94661252699862e2, State: Initialized, Role: FOLLOWER
I20260924 03:08:56.039634 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } }
I20260924 03:08:56.039901 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:56.039857 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } } }
I20260924 03:08:56.039983 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:08:56.041464 7046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:56.041545 7046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:56.042840 6960 master_runner.cc:428] Detected that this master 03bd6c71fc1d465e94661252699862e2 is joining an existing cluster
I20260924 03:08:56.042883 6960 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42329
I20260924 03:08:56.049288 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.043915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.0.98.61:42329)
0924 03:08:56.043945 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.043946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.044037 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:08:56.044101 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.044102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.044109 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.044148 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.044460 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.044869 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.045822 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.046010 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.046056 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.046937 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.046938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.046938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.046943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.048302 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.048657 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.048657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.048657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.048701 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.048885 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.048886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.048886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.048992 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.049040 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.049183 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.049240 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:08:56.049423 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.043773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43412)
0924 03:08:56.043962 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.043978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.044032 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.044161 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.044162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.044327 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.044389 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.044391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.044949 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.044950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.045744 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.045746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.046102 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.046103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.046165 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.046518 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.046522 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.046854 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.048375 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.048376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.048377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.048590 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.048768 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.048769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.048769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.048829 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.049056 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.049057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.049094 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.049170 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.049302 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260924 03:08:56.055033 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.043859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.98.62:37777)
0924 03:08:56.043891 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.043893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.049534 (+ 5641us) server_negotiation.cc:407] Connection header received
0924 03:08:56.049637 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.049639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.049650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.049700 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.049990 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.050796 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.051734 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.051892 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.051936 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.052770 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.052771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.052771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.052777 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.054086 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.054464 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.054465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.054465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.054516 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.054682 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.054682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.054683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.054763 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.054806 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.054933 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.054980 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:08:56.055085 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.043773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58230)
0924 03:08:56.049481 (+ 5708us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.049492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.049558 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.049703 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.049703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.049838 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.049888 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.049889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.050884 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.050885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.051649 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.051652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.051980 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.051982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.052057 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.052370 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.052374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.052683 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.054207 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.054208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.054209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.054394 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.054573 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.054573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.054574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.054627 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.054816 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.054817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.054851 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.054918 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.055001 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5699,"mutex_wait_us":2,"spinlock_wait_cycles":8448,"thread_start_us":94,"threads_started":1}
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:08:56.063694 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.056935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43416)
0924 03:08:56.057060 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.057098 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.057163 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.057292 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.057294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.057499 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.057546 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.057547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.058394 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.058395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.058910 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.058912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.059350 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.059351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.059440 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.059835 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.059838 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.061191 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.062757 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.062762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.062764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.062943 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.063142 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.063142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.063142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.063191 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.063414 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.063419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.063453 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.063526 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.063618 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260924 03:08:56.063735 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.056997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.98.61:42329)
0924 03:08:56.057029 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.057030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.057165 (+ 135us) server_negotiation.cc:407] Connection header received
0924 03:08:56.057213 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.057214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.057222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.057292 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.057612 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.058329 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.058990 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.059276 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.059346 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.061268 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.061269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.061270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.061276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.062674 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.063012 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.063013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.063013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.063066 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.063247 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.063248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.063248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.063356 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.063417 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.063583 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.063661 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:56.064163 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42329: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "03bd6c71fc1d465e94661252699862e2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42329
}
attrs {
promote: true
}
}
dest_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c"
cas_config_opid_index: -1
I20260924 03:08:56.064437 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } }
I20260924 03:08:56.064831 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 03bd6c71fc1d465e94661252699862e2 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } }
I20260924 03:08:56.065037 7002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:56.065693 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42329
I20260924 03:08:56.065672 7002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 03:08:56.065781 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } }
I20260924 03:08:56.065845 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.065989 6960 master.cc:537] Master@127.0.98.61:42329 shutting down...
I20260924 03:08:56.066035 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:08:56.066757 6483 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 03:08:56.067183 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c -> Peer 03bd6c71fc1d465e94661252699862e2 (127.0.98.61:42329): Couldn't send request to peer 03bd6c71fc1d465e94661252699862e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:08:56.068776 6960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:56.068842 6960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:56.068903 6960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: stopping tablet replica
I20260924 03:08:56.081714 6960 master.cc:552] Master@127.0.98.61:42329 shutdown complete.
I20260924 03:08:56.081759 6960 master_runner.cc:305] Clearing existing system tablet
I20260924 03:08:56.081892 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.081979 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.082652 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.083145 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.083281 6960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.083339 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.083364 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "03bd6c71fc1d465e94661252699862e2"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "4827bb8ea7f97af0d0af02ae9bccb4c6"
server_key_iv: "9f5a8fb2190ff16d7af68cf8fabc3638"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.084069 6960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:56.085474 6960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:08:56.085539 6960 log.cc:1196] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 03:08:56.085749 6960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Deleting consensus metadata
I20260924 03:08:56.086632 6960 master_runner.cc:315] Copying system tablet from 127.0.98.62:37777
I20260924 03:08:56.087213 6960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37777
I20260924 03:08:56.092908 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.087537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.0.98.62:37777)
0924 03:08:56.087575 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.087576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.087632 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:08:56.087693 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.087694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.087701 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.087740 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.088043 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.088471 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.089469 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.089646 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.089690 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.090603 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.090604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.090604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.090610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.091943 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.092297 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.092298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.092298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.092341 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.092531 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.092531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.092531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.092614 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.092660 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.092813 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.092859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:08:56.092994 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.087458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58234)
0924 03:08:56.087571 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.087581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.087639 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.087753 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.087753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.087914 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.087980 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.087981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.088541 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.088542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.089389 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.089391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.089710 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.089711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.089774 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.090123 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.090127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.090527 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.092021 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.092022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.092023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.092225 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.092403 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.092404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.092404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.092466 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.092673 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.092673 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.092727 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.092800 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.092886 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":77,"threads_started":1}
I20260924 03:08:56.093271 6516 tablet_copy_service.cc:140] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 03bd6c71fc1d465e94661252699862e2 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58234)
I20260924 03:08:56.093353 6516 tablet_copy_service.cc:161] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 03bd6c71fc1d465e94661252699862e2 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58234: session id = 03bd6c71fc1d465e94661252699862e2-00000000000000000000000000000000
I20260924 03:08:56.094102 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:08:56.094601 6960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:08:56.095350 6960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: tablet copy: Starting download of 0 data blocks...
I20260924 03:08:56.095497 6960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: tablet copy: Starting download of 1 WAL segments...
I20260924 03:08:56.096274 6960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:08:56.096908 6516 tablet_copy_service.cc:342] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Request end of tablet copy session 03bd6c71fc1d465e94661252699862e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58234
I20260924 03:08:56.096961 6516 tablet_copy_service.cc:433] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: ending tablet copy session 03bd6c71fc1d465e94661252699862e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 03bd6c71fc1d465e94661252699862e2
I20260924 03:08:56.099053 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:56.100030 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.100180 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.100185 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.102346 6960 server_base.cc:1034] running on GCE node
I20260924 03:08:56.102432 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.102547 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.103679 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790219336103672 us; error 22 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.104941 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 03:08:56.105168 6960 webserver.cc:460] Webserver started at http://127.0.98.61:41581/ using document root <none> and password file <none>
I20260924 03:08:56.105262 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.105289 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.105842 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:56.106353 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.106474 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.106523 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "03bd6c71fc1d465e94661252699862e2"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "4827bb8ea7f97af0d0af02ae9bccb4c6"
server_key_iv: "9f5a8fb2190ff16d7af68cf8fabc3638"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.106559 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.123394 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.124699 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.127458 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42329
I20260924 03:08:56.127645 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42329 every 8 connection(s)
I20260924 03:08:56.127755 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:08:56.128396 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:08:56.128733 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Bootstrap starting.
I20260924 03:08:56.130836 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:08:56.131076 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Bootstrap complete.
I20260924 03:08:56.131270 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } }
I20260924 03:08:56.131347 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03bd6c71fc1d465e94661252699862e2, State: Initialized, Role: LEARNER
I20260924 03:08:56.131394 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } }
I20260924 03:08:56.131549 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:56.131541 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } }
I20260924 03:08:56.131614 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:08:56.132874 7144 catalog_manager.cc:1259] Loaded cluster ID: 3a5f43f785ce478bb23545154c6fd329
I20260924 03:08:56.132932 7144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: loading cluster ID for follower catalog manager: success
I20260924 03:08:56.133780 7144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: acquiring CA information for follower catalog manager: success
I20260924 03:08:56.134027 7144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:08:56.140789 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.135058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.0.98.61:42329)
0924 03:08:56.135168 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.135169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.135176 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:56.135194 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.135195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.135204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.135289 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.135748 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.136339 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.137018 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.137208 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.137274 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.138335 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.138336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.138336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.138342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.139671 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.140027 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.140028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.140029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.140081 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.140294 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.140296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.140296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.140399 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.140498 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.140619 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.140689 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 03:08:56.166070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34651
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:56.242374 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:56.242552 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:56.242573 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:56.243579 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:56.243613 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:56.243626 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:56.243638 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:56.243649 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:56.244944 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:56.245100 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:56.245245 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:56.249388 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.249375 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.249698 7148 server_base.cc:1034] running on GCE node
W20260924 03:08:56.249375 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.250000 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.250334 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.251483 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790219336251472 us; error 40 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.253854 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 03:08:56.254246 7148 webserver.cc:460] Webserver started at http://127.0.98.60:42551/ using document root <none> and password file <none>
I20260924 03:08:56.254420 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.254458 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.254560 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:56.255527 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/instance:
uuid: "9b8e5dffeff44715a5730627d59b24bf"
format_stamp: "Formatted at 2026-09-24 03:08:56 on dist-test-slave-jtnz"
server_key: "799f393ad669f4ee71b0329c951e0f2b"
server_key_iv: "7419bc2b58a24d23fa7f2da2e5975207"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.255787 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal/instance:
uuid: "9b8e5dffeff44715a5730627d59b24bf"
format_stamp: "Formatted at 2026-09-24 03:08:56 on dist-test-slave-jtnz"
server_key: "799f393ad669f4ee71b0329c951e0f2b"
server_key_iv: "7419bc2b58a24d23fa7f2da2e5975207"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.256902 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:08:56.257457 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.257603 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.257673 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "9b8e5dffeff44715a5730627d59b24bf"
format_stamp: "Formatted at 2026-09-24 03:08:56 on dist-test-slave-jtnz"
server_key: "799f393ad669f4ee71b0329c951e0f2b"
server_key_iv: "7419bc2b58a24d23fa7f2da2e5975207"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.257722 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.278580 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.280130 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:56.280227 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.284097 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34651
I20260924 03:08:56.284121 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34651 every 8 connection(s)
I20260924 03:08:56.284400 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:08:56.285262 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:56.286589 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:08:56.290099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7148
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:56.295156 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.287344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.98.62:37777)
0924 03:08:56.287403 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.287405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.287440 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:08:56.287948 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.287949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.287957 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.288013 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.288525 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.288960 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.289895 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.290119 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.290180 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.292420 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.292421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.292422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.292428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.293776 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.294196 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.294197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.294198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.294260 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.294568 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.294569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.294569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.294669 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.294721 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.295037 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.295096 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:08:56.295400 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.287209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58246)
0924 03:08:56.287342 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.287377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.287853 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.288069 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.288072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.288287 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.288444 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.288447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.289026 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.289026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.289800 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.289802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.290260 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.290262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.290401 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.290929 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.290939 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.292329 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.293873 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.293877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.293879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.294123 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.294322 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.294323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.294338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.294484 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.294736 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.294743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.294897 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.295018 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.295191 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260924 03:08:56.295935 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:08:56.299050 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.290621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.0.98.60:34651)
0924 03:08:56.290741 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.290745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.290762 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:08:56.290781 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.290783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.290816 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.290917 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.292009 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.292796 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.293525 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.293797 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.293906 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.296612 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.296617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.296617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.296622 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.297919 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.298286 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.298289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.298289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.298350 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.298551 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.298552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.298553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.298705 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.298831 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.298936 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.298991 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:08:56.302151 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.296135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43452)
0924 03:08:56.296240 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.296250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.296284 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.296412 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.296412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.296559 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.296608 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.296609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.297451 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.297452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.297890 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.297891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.298331 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.298332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.298400 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.298790 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.298795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.299957 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.301208 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.301210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.301210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.301367 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.301560 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.301561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.301561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.301612 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.301862 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.301862 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.301909 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.301988 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.302071 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:08:56.302199 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.296207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.0.98.61:42329)
0924 03:08:56.296265 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.296267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.296275 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:56.296344 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.296344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.296355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.296395 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.296675 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.297359 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.297967 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.298257 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.298319 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.300037 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.300038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.300039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.300045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.301129 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.301436 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.301437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.301437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.301488 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.301682 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.301683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.301683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.301788 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.301856 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.302018 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.302090 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:08:56.302683 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:08:56.307746 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.302900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:54798)
0924 03:08:56.302946 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.302955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.303010 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.303125 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.303125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.303220 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.303242 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.303243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.303752 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.303753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.304178 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.304178 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.304448 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.304448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.304477 (+ 29us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.304745 (+ 268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.304748 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.305704 (+ 956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.307010 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.307011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.307012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.307153 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.307298 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.307298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.307298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.307341 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.307558 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.307558 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.307586 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.307635 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.307697 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:08:56.307802 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.302956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.98.60:34651)
0924 03:08:56.303005 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.303006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.303013 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:56.303061 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.303061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.303068 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.303122 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.303295 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.303696 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.304233 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.304395 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.304437 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.305771 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.305772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.305772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.305777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.306949 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.307207 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.307208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.307208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.307246 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.307383 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.307384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.307384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.307453 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.307556 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.307652 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.307737 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":35}
I20260924 03:08:56.308881 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Bootstrap starting.
I20260924 03:08:56.309604 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:56.309912 7216 log.cc:824] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:56.310928 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: No bootstrap required, opened a new log
I20260924 03:08:56.312110 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } }
I20260924 03:08:56.312219 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:56.312240 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8e5dffeff44715a5730627d59b24bf, State: Initialized, Role: FOLLOWER
I20260924 03:08:56.312316 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } }
I20260924 03:08:56.312520 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:56.312506 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } } }
I20260924 03:08:56.312597 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:08:56.314049 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:08:56.314119 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:08:56.315960 7148 master_runner.cc:428] Detected that this master 9b8e5dffeff44715a5730627d59b24bf is joining an existing cluster
I20260924 03:08:56.316010 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:34651
I20260924 03:08:56.323493 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.317029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43456)
0924 03:08:56.317194 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.317235 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.317296 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.317480 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.317481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.317675 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.317733 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.317734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318742 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.318743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.319594 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.319597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.320364 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.320364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.320427 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.320836 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.320839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.321089 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.322672 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.322673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.322673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.322818 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.323015 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.323015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.323016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.323069 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.323307 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.323308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.323334 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.323377 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.323435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
I20260924 03:08:56.323539 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.316943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.98.61:42329)
0924 03:08:56.316987 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.316988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.317274 (+ 286us) server_negotiation.cc:407] Connection header received
0924 03:08:56.317349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.317350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.317359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.317426 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.317813 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318633 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319684 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.319996 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.320365 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.321169 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.321170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.321170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.321177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.322603 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.322887 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.322888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.322888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.322943 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.323126 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.323127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.323127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.323249 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.323312 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.323396 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.323460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:08:56.325145 7221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:56.325232 7221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } }
I20260924 03:08:56.325454 7221 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777), 03bd6c71fc1d465e94661252699862e2 (127.0.98.61:42329)
I20260924 03:08:56.325603 6506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b8e5dffeff44715a5730627d59b24bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" is_pre_election: true
I20260924 03:08:56.325634 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b8e5dffeff44715a5730627d59b24bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd6c71fc1d465e94661252699862e2" is_pre_election: true
I20260924 03:08:56.325724 7104 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Handling vote request from an unknown peer 9b8e5dffeff44715a5730627d59b24bf
I20260924 03:08:56.325771 7104 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9b8e5dffeff44715a5730627d59b24bf for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 03:08:56.325955 7167 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8e5dffeff44715a5730627d59b24bf; no voters: 03bd6c71fc1d465e94661252699862e2, 9f19f0db7ee04f8c9ccbc5bf0bf1435c
I20260924 03:08:56.326074 7221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:56.326694 7221 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 03:08:56.337394 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.316872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58250)
0924 03:08:56.317018 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.317027 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.317102 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.317269 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.317269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.317445 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.317530 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.317532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318530 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.318531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.319362 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.319364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.319856 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.319909 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.320324 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.320328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.334443 (+ 14115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.336268 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.336269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.336532 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.336725 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.336726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.336787 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.337058 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.337059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.337097 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.337180 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.337274 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260924 03:08:56.337651 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.317192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.0.98.60:34651)
0924 03:08:56.317222 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.317224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.317424 (+ 200us) server_negotiation.cc:407] Connection header received
0924 03:08:56.317500 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.317501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.317508 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.317531 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.317897 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318641 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319649 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.319924 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319961 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.335019 (+ 15058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.335020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.335021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.335027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.336601 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336938 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.336938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.336939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.336989 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.337168 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.337169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.337169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.337284 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.337355 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.337506 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.337571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:56.337677 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.316899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.98.62:37777)
0924 03:08:56.316941 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.316942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.317091 (+ 149us) server_negotiation.cc:407] Connection header received
0924 03:08:56.317158 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.317159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.317169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.317216 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.317606 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318446 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319463 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.319753 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.319805 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.334557 (+ 14752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.334559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.334559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.334566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.336158 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336615 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.336616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.336617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.336672 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336854 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.336855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.336855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.336983 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.337060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.337504 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.337589 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:56.337867 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.317234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:54808)
0924 03:08:56.317382 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.317390 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.317443 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.317576 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.317576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.317762 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.317812 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.317814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.318748 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.318749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.319579 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.319580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.320053 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.320053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.320113 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.334588 (+ 14475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.334592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.334935 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.336673 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.336674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.336675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.336873 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.337065 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.337065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.337066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.337108 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.337362 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.337362 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.337402 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.337708 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.337778 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:08:56.346081 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.340162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.0.98.60:34651)
0924 03:08:56.340212 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.340214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.340222 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:56.340249 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.340250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.340258 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.340317 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.340563 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.341197 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.341795 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.342092 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.342160 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.343986 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.343987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.343987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.343992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.345142 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.345475 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.345476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.345476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.345521 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.345711 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.345711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.345711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.345797 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.345841 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.345989 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.346035 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:56.346146 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.339585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:54810)
0924 03:08:56.339627 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.339638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.339686 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.340331 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.340332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.340444 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.340489 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.340491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.341269 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.341270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.341730 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.341731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.342163 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.342163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.342255 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.342640 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.342644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.343901 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.345212 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.345213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.345213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.345406 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.345577 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.345577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.345578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.345649 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.345856 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.345857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.345897 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.345972 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.346057 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 03:08:56.346688 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:34651: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 34651
}
attrs {
promote: true
}
}
dest_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c"
cas_config_opid_index: 5
I20260924 03:08:56.346903 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } }
I20260924 03:08:56.347236 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.348212 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:34651
I20260924 03:08:56.348271 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.348327 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.348537 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:08:56.348798 7148 master.cc:537] Master@127.0.98.60:34651 shutting down...
I20260924 03:08:56.348798 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:08:56.349428 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c -> Peer 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60:34651): Couldn't send request to peer 9b8e5dffeff44715a5730627d59b24bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:08:56.351490 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.347351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.98.61:42329)
0924 03:08:56.347383 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.347384 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.347456 (+ 72us) server_negotiation.cc:407] Connection header received
0924 03:08:56.347529 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.347530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.347549 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.347603 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.347734 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.348305 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.349875 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.350496 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.350562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.350641 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.351397 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.351431 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:56.351720 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.347304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:43466)
0924 03:08:56.347406 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.347415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.347466 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.347628 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.347629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.347630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.347660 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.347661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.348395 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.348395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.349781 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.349784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.350673 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.350674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.350734 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.350758 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.351617 (+ 859us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.351661 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:08:56.352008 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:08:56.352661 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:56.352653 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 03:08:56.352732 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:56.352775 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: stopping tablet replica
I20260924 03:08:56.353142 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.354151 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.354254 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:08:56.354223 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: true } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.354332 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:08:56.354905 7055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: attempting to promote NON_VOTER 03bd6c71fc1d465e94661252699862e2 to VOTER
I20260924 03:08:56.355129 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } }
W20260924 03:08:56.355625 6449 connection.cc:535] client connection to 127.0.98.60:34651 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:08:56.355651 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 LEARNER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:08:56.355854 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.355755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:54814)
0924 03:08:56.355794 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.355831 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:08:56.355916 7243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 03:08:56.355995 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c -> Peer 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60:34651): Couldn't send request to peer 9b8e5dffeff44715a5730627d59b24bf. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:08:56.356513 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 03bd6c71fc1d465e94661252699862e2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.357175 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.357192 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.357321 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.357342 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.357334 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 03bd6c71fc1d465e94661252699862e2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.357964 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.358054 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:08:56.366948 7148 master.cc:552] Master@127.0.98.60:34651 shutdown complete.
I20260924 03:08:56.367003 7148 master_runner.cc:305] Clearing existing system tablet
I20260924 03:08:56.367126 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.367172 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.367821 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:56.368312 7245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.368453 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.368510 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.368534 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "9b8e5dffeff44715a5730627d59b24bf"
format_stamp: "Formatted at 2026-09-24 03:08:56 on dist-test-slave-jtnz"
server_key: "799f393ad669f4ee71b0329c951e0f2b"
server_key_iv: "7419bc2b58a24d23fa7f2da2e5975207"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.369212 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:56.370532 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:08:56.370599 7148 log.cc:1196] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 03:08:56.370802 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Deleting consensus metadata
I20260924 03:08:56.370908 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:37777
I20260924 03:08:56.371447 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37777
I20260924 03:08:56.377056 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.371763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.98.62:37777)
0924 03:08:56.371798 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.371800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.371899 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:08:56.371951 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.371952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.371960 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.372013 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.372224 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.372866 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.373605 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.373886 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.373944 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.374722 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.374723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.374723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.374729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.376102 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.376474 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.376475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.376475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.376520 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.376692 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.376693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.376693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.376776 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.376821 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.376955 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.377009 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:08:56.377321 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.371718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:58266)
0924 03:08:56.371841 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.371848 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.371898 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.372014 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.372015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.372128 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.372168 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.372169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.372933 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.372933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.373531 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.373533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.373958 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.373959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.374006 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.374397 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.374401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.374644 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.376178 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.376179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.376180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.376400 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.376576 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.376577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.376577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.376635 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.376830 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.376830 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.376869 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.376940 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.377034 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260924 03:08:56.377545 6516 tablet_copy_service.cc:140] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9b8e5dffeff44715a5730627d59b24bf ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58266)
I20260924 03:08:56.377599 6516 tablet_copy_service.cc:161] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9b8e5dffeff44715a5730627d59b24bf at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58266: session id = 9b8e5dffeff44715a5730627d59b24bf-00000000000000000000000000000000
I20260924 03:08:56.378105 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:08:56.378541 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:08:56.379575 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: tablet copy: Starting download of 0 data blocks...
I20260924 03:08:56.379709 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: tablet copy: Starting download of 1 WAL segments...
I20260924 03:08:56.380375 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:08:56.381286 6516 tablet_copy_service.cc:342] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Request end of tablet copy session 9b8e5dffeff44715a5730627d59b24bf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58266
I20260924 03:08:56.381340 6516 tablet_copy_service.cc:433] P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: ending tablet copy session 9b8e5dffeff44715a5730627d59b24bf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9b8e5dffeff44715a5730627d59b24bf
I20260924 03:08:56.383605 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:56.384603 7252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.384646 7255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.384847 7253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.384866 7148 server_base.cc:1034] running on GCE node
I20260924 03:08:56.384968 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.385083 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.386219 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790219336386208 us; error 25 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.387485 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 03:08:56.387709 7148 webserver.cc:460] Webserver started at http://127.0.98.60:43619/ using document root <none> and password file <none>
I20260924 03:08:56.387833 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.387868 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.388477 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:56.388978 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.389156 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.389209 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "9b8e5dffeff44715a5730627d59b24bf"
format_stamp: "Formatted at 2026-09-24 03:08:56 on dist-test-slave-jtnz"
server_key: "799f393ad669f4ee71b0329c951e0f2b"
server_key_iv: "7419bc2b58a24d23fa7f2da2e5975207"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.389249 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.412688 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.414129 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.417384 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34651
I20260924 03:08:56.417553 7314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34651 every 8 connection(s)
I20260924 03:08:56.417747 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:08:56.418673 7315 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:08:56.419142 7315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Bootstrap starting.
I20260924 03:08:56.421459 7315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:08:56.421715 7315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: Bootstrap complete.
I20260924 03:08:56.421916 7315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } }
I20260924 03:08:56.422027 7315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b8e5dffeff44715a5730627d59b24bf, State: Initialized, Role: LEARNER
I20260924 03:08:56.422072 7315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } }
I20260924 03:08:56.422288 7315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:56.422297 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.422420 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: LEARNER
I20260924 03:08:56.423631 7329 catalog_manager.cc:1259] Loaded cluster ID: 3a5f43f785ce478bb23545154c6fd329
I20260924 03:08:56.423702 7329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: loading cluster ID for follower catalog manager: success
I20260924 03:08:56.424530 7329 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: acquiring CA information for follower catalog manager: success
I20260924 03:08:56.424772 7329 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:08:56.463169 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.456907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.0.98.60:34651)
0924 03:08:56.457040 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.457042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.457049 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:56.457069 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.457070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.457078 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.457161 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.457677 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.458348 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.459042 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.459237 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.459320 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.460566 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.460567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.460567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.460574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.462004 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.462410 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.462411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.462412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.462472 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.462673 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.462674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.462674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.462780 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.462883 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.462979 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.463057 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260924 03:08:56.463629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6558
I20260924 03:08:56.468122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46103
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37777
--webserver_port=36723
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:56.547180 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:56.547366 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:56.547389 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:56.548415 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:56.548447 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:56.548472 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:08:56.549834 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36723
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:56.550009 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:56.550168 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:08:56.554697 7331 server_base.cc:1034] running on GCE node
W20260924 03:08:56.554672 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.554662 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.554672 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.555033 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.555215 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.556375 7331 hybrid_clock.cc:648] HybridClock initialized: now 1790219336556367 us; error 30 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.558616 7331 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:08:56.559013 7331 webserver.cc:460] Webserver started at http://127.0.98.1:36723/ using document root <none> and password file <none>
I20260924 03:08:56.559237 7331 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.559314 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.560505 7331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:56.561003 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.561159 7331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.561261 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "5d6f6aa781794f29af39e98efe90e101"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "bb1831ec44c40ed34d3d861af743f237"
server_key_iv: "453e708d7d22840b82293cb0c4db999d"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.561548 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.582023 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.583936 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:56.584093 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.584308 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:56.584595 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:56.584646 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.584686 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:56.584712 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.590435 7331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46103
I20260924 03:08:56.590487 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46103 every 8 connection(s)
I20260924 03:08:56.590842 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:08:56.594651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7331
I20260924 03:08:56.594791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6692
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:56.601125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43695
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37777
--webserver_port=39695
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:08:56.602625 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47051 (local address 127.0.98.61:42329)
0924 03:08:56.591541 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.591544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.591815 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:08:56.593064 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.593066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.593077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.593118 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.593774 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.594524 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596092 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.596392 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596429 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.598766 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.598768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.598768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.598776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.600601 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.601181 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.601182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.601183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.601242 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.601733 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.601734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.601734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.601900 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.601960 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.602406 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.602491 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:08:56.603116 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41875 (local address 127.0.98.62:37777)
0924 03:08:56.591794 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.591796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.591913 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:08:56.592936 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.592938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.592950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.593024 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.593735 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.594517 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596568 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.596845 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596885 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.599824 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.599826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.599826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.599834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.601801 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.602236 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.602237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.602237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.602295 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.602500 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.602501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.602502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.602643 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.602711 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.602925 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.603016 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:08:56.603320 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:41875)
0924 03:08:56.591860 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.591870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.592819 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.593111 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.593116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.593413 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.593616 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.593620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.594646 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.594646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.596481 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.596483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.596955 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.597031 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.598322 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.598326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.599722 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.601892 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.601893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.601893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.602127 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.602369 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.602369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.602370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.602424 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.602745 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.602746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.602785 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.602907 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.603095 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":9,"thread_start_us":117,"threads_started":1}
I20260924 03:08:56.603717 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.98.1:47051)
0924 03:08:56.591691 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.591733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.592975 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.593193 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.593194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.593469 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.593617 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.593622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.594646 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.594646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.595974 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.595977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.596517 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.596518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.596637 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.597112 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.597121 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.598651 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.600722 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.600729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.600731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.601084 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.601519 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.601520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.601521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.601650 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.602059 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.602066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.602248 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.602930 (+ 682us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.603592 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":645,"thread_start_us":96,"threads_started":1}
I20260924 03:08:56.603761 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:56.603953 7461 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.604161 7461 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
I20260924 03:08:56.604485 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42329
I20260924 03:08:56.604539 7462 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.604526 6496 ts_manager.cc:194] Re-registered known tserver with Master: 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
I20260924 03:08:56.604593 7462 heartbeater.cc:511] Master 127.0.98.61:42329 requested a full tablet report, sending...
I20260924 03:08:56.604970 7094 ts_manager.cc:194] Registered new tserver with Master: 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
I20260924 03:08:56.604991 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41875
I20260924 03:08:56.606045 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.98.1:44171)
0924 03:08:56.591930 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.591939 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.592821 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.593110 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.593113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.593470 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.593617 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.593622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.597986 (+ 4364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.597987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.598930 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.598932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.599510 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.599511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.599584 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.600758 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.600763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.602322 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.605121 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.605122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.605123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.605303 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.605485 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.605485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.605486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.605539 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.605800 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.605800 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.605847 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.605905 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.605974 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260924 03:08:56.606163 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.591853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44171 (local address 127.0.98.60:34651)
0924 03:08:56.591999 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.592001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.592012 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:56.592934 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.592935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.592949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.593016 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.594261 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.594888 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.599035 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.599379 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.599431 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.602429 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.602431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.602431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.602438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.605018 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.605354 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.605355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.605356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.605416 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.605597 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.605597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.605598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.605741 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.605805 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.605949 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.606036 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 03:08:56.606531 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34651
I20260924 03:08:56.606599 7460 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.606667 7460 heartbeater.cc:511] Master 127.0.98.60:34651 requested a full tablet report, sending...
I20260924 03:08:56.607081 7279 ts_manager.cc:194] Registered new tserver with Master: 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
W20260924 03:08:56.681656 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:56.681828 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:56.681849 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:56.682976 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:56.683009 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:56.683034 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:08:56.684401 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39695
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:56.684610 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:56.684840 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:56.689513 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.689523 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.689531 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.689764 7467 server_base.cc:1034] running on GCE node
I20260924 03:08:56.690060 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.690315 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.691469 7467 hybrid_clock.cc:648] HybridClock initialized: now 1790219336691461 us; error 25 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.693646 7467 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:08:56.694104 7467 webserver.cc:460] Webserver started at http://127.0.98.2:39695/ using document root <none> and password file <none>
I20260924 03:08:56.694334 7467 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.694410 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.695573 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:56.696103 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.696256 7467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:56.696344 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "d2114174f5a1065932eee340ff86eff4"
server_key_iv: "37af3ddfa0d68f1a84888090ab4eefb4"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.696636 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.723982 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.725557 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:56.725694 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.725878 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:56.726205 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:56.726246 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.726279 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:56.726302 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.732118 7467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43695
I20260924 03:08:56.732161 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43695 every 8 connection(s)
I20260924 03:08:56.732494 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:08:56.738173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7467
I20260924 03:08:56.738340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6826
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:56.742372 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50381 (local address 127.0.98.62:37777)
0924 03:08:56.733769 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.733771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.733818 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:08:56.734019 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.734021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.734032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.734063 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.734754 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.735231 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736542 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.736781 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736854 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.739638 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.739640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.739640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.739648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.740916 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.741311 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.741312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.741312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.741372 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.741618 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.741619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.741620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.741733 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.741810 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.742121 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.742229 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:56.742578 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:50381)
0924 03:08:56.733770 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.733778 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.733928 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.734142 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.734143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.734491 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.734647 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.734652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.735322 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.735323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.736464 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.736466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.736859 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.737010 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.738157 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.738163 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.739547 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.741018 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.741024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.741026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.741235 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.741444 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.741445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.741445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.741552 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.741801 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.741807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.741946 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.742089 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.742288 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260924 03:08:56.743084 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47139 (local address 127.0.98.61:42329)
0924 03:08:56.733531 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.733533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.733650 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:08:56.734055 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.734056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.734066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.734111 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.734783 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.735469 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736634 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.736922 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736988 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.739376 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.739378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.739378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.739385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.741634 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.742046 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.742048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.742048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.742109 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.742342 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.742343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.742343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.742672 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.742719 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.742928 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.742996 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:08:56.743095 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.98.2:47139)
0924 03:08:56.733594 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.733603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.733926 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.734510 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.734511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.734640 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.734687 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.734689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.735557 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.735557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.736491 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.736493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.737006 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.737007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.737058 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.737551 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.737560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.739270 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.741728 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.741729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.741729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.741971 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.742202 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.742203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.742203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.742265 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.742792 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.742793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.742834 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.742890 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.742984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":50,"thread_start_us":66,"threads_started":1}
I20260924 03:08:56.744313 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:56.744414 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42329
I20260924 03:08:56.744405 7597 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.744509 7598 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.744668 7597 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
I20260924 03:08:56.744674 7598 heartbeater.cc:511] Master 127.0.98.61:42329 requested a full tablet report, sending...
I20260924 03:08:56.745059 7094 ts_manager.cc:194] Registered new tserver with Master: 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
I20260924 03:08:56.745651 6496 ts_manager.cc:194] Re-registered known tserver with Master: 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
I20260924 03:08:56.746009 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50381
I20260924 03:08:56.746426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46515
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37777
--webserver_port=40315
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:08:56.749763 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46587 (local address 127.0.98.60:34651)
0924 03:08:56.733186 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.733187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.733381 (+ 194us) server_negotiation.cc:407] Connection header received
0924 03:08:56.734006 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.734007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.734018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.734065 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.734739 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.735469 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.737299 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.737577 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.737650 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.747474 (+ 9824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.747476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.747476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.747484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.748666 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.748988 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.748989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.748990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.749050 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.749237 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.749238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.749239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.749345 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.749419 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.749594 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.749670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:08:56.749787 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.733139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.98.2:46587)
0924 03:08:56.733267 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.733295 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.733905 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.734137 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.734141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.734469 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.734646 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.734650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.736459 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.736459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.737227 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.737229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.737658 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.737658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.737705 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.746227 (+ 8522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.746233 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.747383 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.748745 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.748746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.748747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.748906 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.749113 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.749114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.749114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.749167 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.749408 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.749409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.749480 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.749570 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.749670 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260924 03:08:56.750375 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34651
I20260924 03:08:56.750427 7596 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.750469 7596 heartbeater.cc:511] Master 127.0.98.60:34651 requested a full tablet report, sending...
I20260924 03:08:56.750759 7279 ts_manager.cc:194] Registered new tserver with Master: 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
W20260924 03:08:56.823158 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:56.823347 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:56.823381 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:56.824520 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:56.824559 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:56.824584 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:08:56.825934 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40315
--tserver_master_addrs=127.0.98.62:37777,127.0.98.61:42329,127.0.98.60:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:56.826160 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:56.826423 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:56.830933 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.830942 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:56.831069 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:56.831161 7603 server_base.cc:1034] running on GCE node
I20260924 03:08:56.831383 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:56.831630 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:56.832782 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790219336832772 us; error 37 us; skew 500 ppm
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:56.835140 7603 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:08:56.835564 7603 webserver.cc:460] Webserver started at http://127.0.98.3:40315/ using document root <none> and password file <none>
I20260924 03:08:56.835803 7603 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:56.835851 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:56.837045 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:56.837561 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.837730 7603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:56.837812 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "ee73197bf4814bdf8180f1ace4605ccf"
format_stamp: "Formatted at 2026-09-24 03:08:55 on dist-test-slave-jtnz"
server_key: "33983e4c1b63853c9f1d12061cf28c87"
server_key_iv: "431d80925dbaebaaab8fb9548d283876"
server_key_version: "encryptionkey@0"
I20260924 03:08:56.838119 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:56.848511 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:56.850639 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:56.850775 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:56.850972 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:08:56.851228 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:08:56.851276 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.851313 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:08:56.851338 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:56.856637 7603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46515
I20260924 03:08:56.856693 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46515 every 8 connection(s)
I20260924 03:08:56.856978 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:08:56.860873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7603
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 03:08:56 dist-test-slave-jtnz krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:56.870229 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.857711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38691 (local address 127.0.98.62:37777)
0924 03:08:56.857758 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.857759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.857958 (+ 199us) server_negotiation.cc:407] Connection header received
0924 03:08:56.858981 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.858983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.858995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.859028 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.859757 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860512 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.862591 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.862876 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.862916 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.865958 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.865960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.865960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.865967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.867748 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.868956 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.868958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.868958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.869033 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.869310 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.869311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.869312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.869459 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.869543 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.869984 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.870073 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:08:56.870245 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.857533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:38691)
0924 03:08:56.857907 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.857917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.858885 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.859115 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.859118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.859484 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.859653 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.859658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860611 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.860611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.862497 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.862499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.862996 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.862997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.863061 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.863433 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.863439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.865872 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.867831 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.867836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.867838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.868148 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.869113 (+ 965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.869114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.869114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.869238 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.869571 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.869577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.869720 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.869831 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.869998 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":6,"thread_start_us":195,"threads_started":1}
I20260924 03:08:56.870680 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37777
I20260924 03:08:56.870770 7733 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.870961 7733 heartbeater.cc:511] Master 127.0.98.62:37777 requested a full tablet report, sending...
I20260924 03:08:56.871357 6496 ts_manager.cc:194] Re-registered known tserver with Master: ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3:46515)
I20260924 03:08:56.871487 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.858219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39569 (local address 127.0.98.61:42329)
0924 03:08:56.858268 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.858270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.858279 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:56.858991 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.858992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.859003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.859032 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.859757 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860504 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.861673 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.861956 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.862044 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.866017 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.866018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.866018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.866025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.868874 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.870659 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.870660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.870660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.870726 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.870923 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.870924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.870924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.871060 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.871112 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.871283 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.871376 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:08:56.871765 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.857977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.98.3:39569)
0924 03:08:56.858122 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.858132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.858885 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.859115 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.859118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.859490 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.859653 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.859658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860611 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.860611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.861590 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.861592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.862046 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.862047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.862114 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.863736 (+ 1622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.863742 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.865915 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.870358 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.870359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.870360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.870568 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.870798 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.870799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.870799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.870853 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.871144 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.871145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.871185 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.871315 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.871663 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":11,"thread_start_us":78,"threads_started":1}
I20260924 03:08:56.871809 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38691
I20260924 03:08:56.872201 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42329
I20260924 03:08:56.872251 7734 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.872419 7734 heartbeater.cc:511] Master 127.0.98.61:42329 requested a full tablet report, sending...
I20260924 03:08:56.872680 7094 ts_manager.cc:194] Registered new tserver with Master: ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3:46515)
I20260924 03:08:56.873359 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.857652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47055 (local address 127.0.98.60:34651)
0924 03:08:56.857728 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.857730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.857928 (+ 198us) server_negotiation.cc:407] Connection header received
0924 03:08:56.858976 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.858978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.858990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.859050 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.859748 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860224 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.861319 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.861505 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.861583 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.867930 (+ 6347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.867931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.867931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.867938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.869673 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.871142 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.871143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.871143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.871195 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.872839 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.872840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.872841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.872965 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.873019 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.873206 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.873271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:08:56.873685 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.857533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.98.3:47055)
0924 03:08:56.857821 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.857858 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.858868 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.859114 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.859117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.859456 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:08:56.859652 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.859656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.860324 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.860325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.861230 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.861232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.861592 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.861593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.861724 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.862339 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:08:56.862347 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:08:56.867832 (+ 5485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:08:56.869775 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.869776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.869776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.870380 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.872713 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:08:56.872714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:08:56.872714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:08:56.872770 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:08:56.873052 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:08:56.873052 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:08:56.873089 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.873478 (+ 389us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.873569 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":323,"threads_started":1}
I20260924 03:08:56.873950 7732 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34651
I20260924 03:08:56.873992 7732 heartbeater.cc:464] Registering TS with master...
I20260924 03:08:56.874076 7732 heartbeater.cc:511] Master 127.0.98.60:34651 requested a full tablet report, sending...
I20260924 03:08:56.874356 7279 ts_manager.cc:194] Registered new tserver with Master: ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3:46515)
I20260924 03:08:56.874913 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.862840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.98.60:34651)
0924 03:08:56.863217 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.863219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.863519 (+ 300us) server_negotiation.cc:407] Connection header received
0924 03:08:56.863614 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.863615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.863627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.863663 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.864867 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.865585 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.866634 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.866918 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.866963 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.868369 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.868370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.868370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.868376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.872631 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.874121 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.874122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.874122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.874175 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.874374 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.874375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.874375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.874544 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.874654 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.874755 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.874827 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":44,"threads_started":1}
I20260924 03:08:56.875195 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.862492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.98.61:42329)
0924 03:08:56.862607 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.862609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.862833 (+ 224us) server_negotiation.cc:407] Connection header received
0924 03:08:56.862933 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.862934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.862946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.862990 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.864225 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.865052 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.866116 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.867536 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.867580 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.871207 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.871208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.871208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.871214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.873355 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.873738 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.873739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.873739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.873787 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.874263 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.874264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.874264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.874380 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.874458 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.874836 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.874920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 03:08:56.875372 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.863196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.98.62:37777)
0924 03:08:56.863280 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.863281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.864484 (+ 1203us) server_negotiation.cc:407] Connection header received
0924 03:08:56.864577 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.864578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.864590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.864632 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:56.865339 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.866794 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.868217 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.868859 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.868926 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.871389 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:56.871390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:56.871390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:56.871395 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:56.873245 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.873653 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.873654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.873655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.873703 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:56.874642 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:56.874643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:56.874643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:56.874758 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:56.874836 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.875210 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.875284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":49,"threads_started":1}
I20260924 03:08:56.876654 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:56.878247 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:56.881040 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.877845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:46416)
0924 03:08:56.877923 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.877932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.877989 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.878129 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.878130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.878132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.878174 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.878195 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.878876 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.878876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.879912 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.879915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.880786 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.880787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.880855 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.880860 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.880912 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.880947 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":48,"threads_started":1}
I20260924 03:08:56.881212 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.877901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.0.98.60:34651)
0924 03:08:56.877935 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.877937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.877971 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:08:56.878026 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.878027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.878046 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.878122 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.878277 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.878811 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.879978 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.880535 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.880614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.880706 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.881122 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.881158 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:56.881474 7289 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 03:08:56.881623 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.881703 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: LEARNER
I20260924 03:08:56.881987 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: true } } }
I20260924 03:08:56.882087 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.883240 7752 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: attempting to promote NON_VOTER 9b8e5dffeff44715a5730627d59b24bf to VOTER
I20260924 03:08:56.883358 7752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } }
I20260924 03:08:56.883790 7289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 LEARNER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 03:08:56.883947 7243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 03:08:56.884016 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Refusing update from remote peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 03:08:56.884549 7752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 03:08:56.885016 7243 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.885865 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.885931 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.886039 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.881619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43695 (local address 127.0.0.1:37378)
0924 03:08:56.881895 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.881922 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.881960 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.882424 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.882425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.882426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.882455 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.882456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.883749 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.883750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.884815 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.884818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.885845 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.885846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.885899 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.885902 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.885926 (+ 24us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.885968 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":69,"threads_started":1}
I20260924 03:08:56.886144 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.885965 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.886390 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.886678 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.881793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.0.1:43338)
0924 03:08:56.881922 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.881931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.881988 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.882644 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.882645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.882646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.882685 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.882687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.884326 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.884327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.885396 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.885398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.886428 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.886429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.886478 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.886481 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.886536 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.886594 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260924 03:08:56.886890 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.882088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.0.98.2:43695)
0924 03:08:56.882262 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.882266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.882277 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:08:56.882311 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.882313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.882327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.882517 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.882569 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.883661 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.885018 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.885761 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.886490 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.886590 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.886627 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.886679 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":293,"threads_started":1}
I20260924 03:08:56.887274 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.882238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.0.98.1:46103)
0924 03:08:56.882433 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.882437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.882479 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:08:56.882531 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.882533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.882548 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.882776 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.882830 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.884232 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.885512 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.886339 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.886779 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.887115 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.887145 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.887186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260924 03:08:56.887666 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.881603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.0.1:54796)
0924 03:08:56.881724 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.881734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.881773 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.883049 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.883050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.883052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.883086 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.883087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.885472 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.885473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.886473 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.886476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.887476 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.887478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.887538 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.887542 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.887572 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.887609 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 03:08:56.887571 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 (DEFAULT_TABLE table=test-table [id=91e4d8e96a724aa9a50b34d39b7bae71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:56.887920 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11330b01dfaa4abebe708b8ae2af5303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:56.887873 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 (DEFAULT_TABLE table=test-table [id=91e4d8e96a724aa9a50b34d39b7bae71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:56.887991 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.881679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.0.98.3:46515)
0924 03:08:56.882867 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.882870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.882880 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:08:56.882925 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.882927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.882940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.883168 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.884179 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.885154 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.886643 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.887368 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.887804 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.887874 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.887898 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.887930 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":66,"threads_started":1}
I20260924 03:08:56.888046 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.888113 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:56.888163 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11330b01dfaa4abebe708b8ae2af5303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:56.888306 7289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.888468 7666 tablet_service.cc:1467] Processing CreateTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 (DEFAULT_TABLE table=test-table [id=91e4d8e96a724aa9a50b34d39b7bae71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:56.888743 7666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11330b01dfaa4abebe708b8ae2af5303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:56.889235 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.889300 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:08:56.891322 7767 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Bootstrap starting.
I20260924 03:08:56.891708 7768 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Bootstrap starting.
I20260924 03:08:56.891824 7767 tablet_bootstrap.cc:654] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:56.892356 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:56.892454 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:08:56.892529 7767 log.cc:824] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:56.892584 7769 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Bootstrap starting.
I20260924 03:08:56.892668 7768 tablet_bootstrap.cc:654] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:56.892912 7768 log.cc:824] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:56.893122 7769 tablet_bootstrap.cc:654] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:56.893378 7767 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: No bootstrap required, opened a new log
I20260924 03:08:56.893383 7769 log.cc:824] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:56.893447 7767 ts_tablet_manager.cc:1397] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:56.893574 7768 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: No bootstrap required, opened a new log
I20260924 03:08:56.893631 7768 ts_tablet_manager.cc:1397] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:56.894032 7769 tablet_bootstrap.cc:492] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: No bootstrap required, opened a new log
I20260924 03:08:56.894101 7769 ts_tablet_manager.cc:1397] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:08:56.895349 7768 raft_consensus.cc:348] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.895411 7767 raft_consensus.cc:348] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.895521 7767 raft_consensus.cc:374] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:56.895521 7768 raft_consensus.cc:374] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:56.895553 7767 raft_consensus.cc:729] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6f6aa781794f29af39e98efe90e101, State: Initialized, Role: FOLLOWER
I20260924 03:08:56.895553 7768 raft_consensus.cc:729] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad6f36e5aed4b09a938f7897a768ddb, State: Initialized, Role: FOLLOWER
I20260924 03:08:56.895668 7767 consensus_queue.cc:260] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.895658 7768 consensus_queue.cc:260] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.895720 7769 raft_consensus.cc:348] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.895860 7769 raft_consensus.cc:374] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:56.895889 7767 ts_tablet_manager.cc:1428] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:08:56.895889 7768 ts_tablet_manager.cc:1428] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:08:56.895901 7769 raft_consensus.cc:729] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee73197bf4814bdf8180f1ace4605ccf, State: Initialized, Role: FOLLOWER
I20260924 03:08:56.895984 7597 heartbeater.cc:503] Master 127.0.98.62:37777 was elected leader, sending a full tablet report...
I20260924 03:08:56.896023 7769 consensus_queue.cc:260] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.896234 7769 ts_tablet_manager.cc:1428] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:08:56.896543 7733 heartbeater.cc:503] Master 127.0.98.62:37777 was elected leader, sending a full tablet report...
I20260924 03:08:56.897017 7461 heartbeater.cc:503] Master 127.0.98.62:37777 was elected leader, sending a full tablet report...
I20260924 03:08:56.968086 7776 raft_consensus.cc:482] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:56.968181 7776 raft_consensus.cc:504] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.968555 7776 leader_election.cc:288] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695), 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
I20260924 03:08:56.972144 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.968830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52211 (local address 127.0.98.1:46103)
0924 03:08:56.968866 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.968867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.968874 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:56.968913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.968914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.968923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.968980 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.969152 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.969897 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971336 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.971805 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971870 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.971879 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.972049 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.972083 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:08:56.972144 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.968741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33325 (local address 127.0.98.2:43695)
0924 03:08:56.968780 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.968782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.968792 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:08:56.968859 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.968860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.968873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.968977 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:56.969153 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.969894 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971336 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.971805 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971870 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.971879 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.972049 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.972081 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:08:56.972179 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.968709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.98.3:52211)
0924 03:08:56.968753 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.968764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.968801 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.968987 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.968988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.968989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.969077 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.969078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.969973 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.969974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.971257 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.971260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.971875 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.971972 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.971976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.972034 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.972071 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":82}
I20260924 03:08:56.972188 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43695 (local address 127.0.98.3:33325)
0924 03:08:56.968753 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:56.968764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:56.968801 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:56.968987 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:56.968988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:56.968989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:56.969077 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.969078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.969985 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.969985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.971257 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:56.971260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.971875 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.971876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:56.971972 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.971976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:56.972029 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:08:56.972071 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":81}
I20260924 03:08:56.972560 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11330b01dfaa4abebe708b8ae2af5303" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" is_pre_election: true
I20260924 03:08:56.972718 7550 raft_consensus.cc:2454] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 0.
I20260924 03:08:56.972718 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11330b01dfaa4abebe708b8ae2af5303" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d6f6aa781794f29af39e98efe90e101" is_pre_election: true
I20260924 03:08:56.972820 7414 raft_consensus.cc:2454] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 0.
I20260924 03:08:56.972934 7619 leader_election.cc:302] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad6f36e5aed4b09a938f7897a768ddb, ee73197bf4814bdf8180f1ace4605ccf; no voters:
I20260924 03:08:56.973053 7776 raft_consensus.cc:2781] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:56.973124 7776 raft_consensus.cc:482] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:56.973158 7776 raft_consensus.cc:3037] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:56.973763 7776 raft_consensus.cc:504] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.973892 7776 leader_election.cc:288] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 election: Requested vote from peers 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695), 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103)
I20260924 03:08:56.974048 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11330b01dfaa4abebe708b8ae2af5303" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
I20260924 03:08:56.974119 7550 raft_consensus.cc:3037] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:56.974815 7550 raft_consensus.cc:2454] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 1.
I20260924 03:08:56.974927 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11330b01dfaa4abebe708b8ae2af5303" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d6f6aa781794f29af39e98efe90e101"
I20260924 03:08:56.974958 7619 leader_election.cc:302] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad6f36e5aed4b09a938f7897a768ddb, ee73197bf4814bdf8180f1ace4605ccf; no voters:
I20260924 03:08:56.975003 7414 raft_consensus.cc:3037] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:56.975034 7776 raft_consensus.cc:2781] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:56.975174 7776 raft_consensus.cc:686] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 1 LEADER]: Becoming Leader. State: Replica: ee73197bf4814bdf8180f1ace4605ccf, State: Running, Role: LEADER
I20260924 03:08:56.975366 7776 consensus_queue.cc:237] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } }
I20260924 03:08:56.975854 7414 raft_consensus.cc:2454] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 1.
I20260924 03:08:56.976189 6496 catalog_manager.cc:5495] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ee73197bf4814bdf8180f1ace4605ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } health_report { overall_health: HEALTHY } } }
W20260924 03:08:56.983139 7599 tablet.cc:2369] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:56.999993 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:56.997678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.98.3:46515)
0924 03:08:56.997729 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:56.997730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:56.997738 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:56.997766 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:56.997766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:56.997782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:56.997843 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:56.997974 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.998477 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.999254 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:56.999451 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:56.999541 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:56.999695 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:56.999788 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:56.999877 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:56.999902 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:08:56.999921 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:08:57.001886 7550 raft_consensus.cc:1261] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Refusing update from remote peer ee73197bf4814bdf8180f1ace4605ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:57.001993 7414 raft_consensus.cc:1261] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Refusing update from remote peer ee73197bf4814bdf8180f1ace4605ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:57.002240 7776 consensus_queue.cc:1035] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:57.002385 7777 consensus_queue.cc:1035] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:57.005895 7781 mvcc.cc:205] Tried to move back new op lower bound from 7332738404357574656 to 7332738404251803648. Current Snapshot: MvccSnapshot[applied={T|T < 7332738404357574656 or (T in {7332738404357574656})}]
I20260924 03:08:57.006016 7782 mvcc.cc:205] Tried to move back new op lower bound from 7332738404357574656 to 7332738404251803648. Current Snapshot: MvccSnapshot[applied={T|T < 7332738404357574656}]
I20260924 03:08:57.006901 6495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:57.006985 6495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:57.008283 6495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f19f0db7ee04f8c9ccbc5bf0bf1435c: Sending DeleteTablet for 3 replicas of tablet 11330b01dfaa4abebe708b8ae2af5303
I20260924 03:08:57.008581 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37378
I20260924 03:08:57.008590 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43338
I20260924 03:08:57.008808 7666 tablet_service.cc:1514] Processing DeleteTablet for tablet 11330b01dfaa4abebe708b8ae2af5303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54796
I20260924 03:08:57.009055 7792 tablet_replica.cc:330] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: stopping tablet replica
I20260924 03:08:57.009147 7792 raft_consensus.cc:2229] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:57.009230 7792 raft_consensus.cc:2258] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:57.009418 7790 tablet_replica.cc:330] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: stopping tablet replica
I20260924 03:08:57.009531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6464
I20260924 03:08:57.009747 7792 ts_tablet_manager.cc:1905] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:57.011653 7792 ts_tablet_manager.cc:1918] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:57.011721 7792 log.cc:1196] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/11330b01dfaa4abebe708b8ae2af5303
I20260924 03:08:57.011950 7792 ts_tablet_manager.cc:1939] T 11330b01dfaa4abebe708b8ae2af5303 P ee73197bf4814bdf8180f1ace4605ccf: Deleting consensus metadata
I20260924 03:08:57.012310 7791 tablet_replica.cc:330] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: stopping tablet replica
I20260924 03:08:57.012401 7791 raft_consensus.cc:2229] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:57.012490 7791 raft_consensus.cc:2258] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:57.012871 7791 ts_tablet_manager.cc:1905] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:57.014277 7790 raft_consensus.cc:2229] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:57.014420 7790 raft_consensus.cc:2258] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:57.014576 7791 ts_tablet_manager.cc:1918] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:57.014643 7791 log.cc:1196] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/11330b01dfaa4abebe708b8ae2af5303
I20260924 03:08:57.014864 7791 ts_tablet_manager.cc:1939] T 11330b01dfaa4abebe708b8ae2af5303 P 5d6f6aa781794f29af39e98efe90e101: Deleting consensus metadata
W20260924 03:08:57.016243 7347 connection.cc:535] server connection from 127.0.0.1:43338 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 03:08:57.016371 7622 connection.cc:535] server connection from 127.0.0.1:54796 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:08:57.018411 7790 ts_tablet_manager.cc:1905] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:57.019872 7790 ts_tablet_manager.cc:1918] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:57.019942 7790 log.cc:1196] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/11330b01dfaa4abebe708b8ae2af5303
I20260924 03:08:57.020185 7790 ts_tablet_manager.cc:1939] T 11330b01dfaa4abebe708b8ae2af5303 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting consensus metadata
W20260924 03:08:57.020475 7485 connection.cc:441] server connection from 127.0.0.1:37378 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37378 (request call id 1) could send its response
I20260924 03:08:57.025092 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:57.017713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.61:42329)
0924 03:08:57.017748 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:57.017749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:57.017893 (+ 144us) server_negotiation.cc:407] Connection header received
0924 03:08:57.017991 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:57.017992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:57.018003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:57.018030 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:57.018666 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.019266 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.020352 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.020665 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.020701 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:57.022258 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:57.022259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:57.022259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:57.022266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:57.023892 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.024260 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.024261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.024261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.024314 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.024527 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.024528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.024529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.024651 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:57.024720 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:57.024907 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:08:57.025004 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:08:57.025180 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:57.017831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.98.60:34651)
0924 03:08:57.017862 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:57.017864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:57.017939 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:08:57.017989 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:57.017990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:57.017998 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:57.018047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:57.018765 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.019444 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.020645 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:57.020933 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:57.020980 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:57.022356 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:57.022358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:57.022358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:57.022365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:57.024051 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.024434 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.024435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.024435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.024484 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:57.024699 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:57.024699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:57.024700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:57.024804 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:57.024900 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:57.025007 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:08:57.025074 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:08:58.006059 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.005777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:57713)
0924 03:08:58.005898 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.005976 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
W20260924 03:08:58.006327 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.006829 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.006577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:50283)
0924 03:08:58.006732 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.006788 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260924 03:08:58.007216 7461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.007321 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.007104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:45151)
0924 03:08:58.007204 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.007256 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":44,"threads_started":1}
W20260924 03:08:58.007460 7733 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.508486 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c)
I20260924 03:08:58.508603 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } }
I20260924 03:08:58.508703 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c)
I20260924 03:08:58.508790 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } }
I20260924 03:08:58.508953 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777), 03bd6c71fc1d465e94661252699862e2 (127.0.98.61:42329)
I20260924 03:08:58.509045 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777), 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60:34651)
I20260924 03:08:58.509230 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.508965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37604)
0924 03:08:58.509142 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.509188 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":48,"threads_started":1}
I20260924 03:08:58.509234 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.509032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37620)
0924 03:08:58.509145 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.509193 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":45,"threads_started":1}
W20260924 03:08:58.509342 7079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
W20260924 03:08:58.509488 7264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.509825 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.509746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37634)
0924 03:08:58.509781 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.509802 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 03:08:58.509920 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.510169 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.510065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37636)
0924 03:08:58.510121 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.510143 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 03:08:58.510303 7264 leader_election.cc:334] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.512543 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.508972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34651 (local address 127.0.0.1:46436)
0924 03:08:58.509289 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.509297 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.509346 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.509532 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.509533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.509536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:58.509631 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.509632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.510578 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.510579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.511673 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.511676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.512331 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.512331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.512375 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.512378 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.512435 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.512465 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":48,"threads_started":1}
I20260924 03:08:58.512581 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.509190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.98.60:34651)
0924 03:08:58.509384 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.509386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.509394 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:58.509418 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.509419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.509428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.509513 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:58.509725 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.510497 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.511746 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.512235 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.512306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.512329 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.512468 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.512510 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":40,"threads_started":1}
I20260924 03:08:58.512737 7289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03bd6c71fc1d465e94661252699862e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b8e5dffeff44715a5730627d59b24bf" is_pre_election: true
I20260924 03:08:58.512830 7289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03bd6c71fc1d465e94661252699862e2 in term 1.
I20260924 03:08:58.512971 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03bd6c71fc1d465e94661252699862e2, 9b8e5dffeff44715a5730627d59b24bf; no voters: 9f19f0db7ee04f8c9ccbc5bf0bf1435c
I20260924 03:08:58.513074 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.509176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.98.61:42329)
0924 03:08:58.509338 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.509340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.509713 (+ 373us) server_negotiation.cc:407] Connection header received
0924 03:08:58.509736 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.509737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.509746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.509803 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:58.510114 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.510727 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.511889 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.512474 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.512536 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.512546 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.512968 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.513012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":58,"threads_started":1}
I20260924 03:08:58.513077 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 03:08:58.513144 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f19f0db7ee04f8c9ccbc5bf0bf1435c)
I20260924 03:08:58.513198 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:08:58.513854 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } }
I20260924 03:08:58.513974 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 election: Requested vote from peers 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777), 9b8e5dffeff44715a5730627d59b24bf (127.0.98.60:34651)
I20260924 03:08:58.514070 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.509125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42329 (local address 127.0.0.1:54658)
0924 03:08:58.509263 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.509535 (+ 272us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.509619 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.509996 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.509997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.509999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:58.510034 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.510036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.510798 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.510799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.511825 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.511828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.512548 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.512549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.512687 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.512692 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.512951 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.513000 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":40,"threads_started":1}
I20260924 03:08:58.514091 7289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03bd6c71fc1d465e94661252699862e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b8e5dffeff44715a5730627d59b24bf"
I20260924 03:08:58.514134 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.514030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37644)
0924 03:08:58.514094 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.514115 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:08:58.514153 7289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:08:58.514262 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b8e5dffeff44715a5730627d59b24bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03bd6c71fc1d465e94661252699862e2" is_pre_election: true
I20260924 03:08:58.514328 7104 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b8e5dffeff44715a5730627d59b24bf in current term 2: Already voted for candidate 03bd6c71fc1d465e94661252699862e2 in this term.
I20260924 03:08:58.514391 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.514316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37652)
0924 03:08:58.514353 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.514378 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:08:58.514439 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8e5dffeff44715a5730627d59b24bf; no voters: 03bd6c71fc1d465e94661252699862e2, 9f19f0db7ee04f8c9ccbc5bf0bf1435c
W20260924 03:08:58.514456 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
I20260924 03:08:58.514894 7289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03bd6c71fc1d465e94661252699862e2 in term 2.
I20260924 03:08:58.515025 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:08:58.515101 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03bd6c71fc1d465e94661252699862e2, 9b8e5dffeff44715a5730627d59b24bf; no voters: 9f19f0db7ee04f8c9ccbc5bf0bf1435c
I20260924 03:08:58.515187 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:08:58.515316 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [term 2 LEADER]: Becoming Leader. State: Replica: 03bd6c71fc1d465e94661252699862e2, State: Running, Role: LEADER
I20260924 03:08:58.515384 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } }
I20260924 03:08:58.515694 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bd6c71fc1d465e94661252699862e2. Latest consensus state: current_term: 2 leader_uuid: "03bd6c71fc1d465e94661252699862e2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:58.515774 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:58.515944 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:58.516338 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:58.516419 7815 catalog_manager.cc:1259] Loaded cluster ID: 3a5f43f785ce478bb23545154c6fd329
I20260924 03:08:58.516448 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:08:58.516599 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:58.516687 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Loaded TSK: 0
I20260924 03:08:58.516808 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:08:58.548350 7093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:08:58.549806 7093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:08:58.550843 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.550739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37676)
0924 03:08:58.550794 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.550824 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:08:58.550856 7289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [term 2 FOLLOWER]: Refusing update from remote peer 03bd6c71fc1d465e94661252699862e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 03:08:58.551069 7803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 03:08:58.551210 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.551117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.0.1:37686)
0924 03:08:58.551161 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.551187 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:08:58.551301 7079 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 -> Peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c (127.0.98.62:37777): Couldn't send request to peer 9f19f0db7ee04f8c9ccbc5bf0bf1435c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:08:58.551625 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bd6c71fc1d465e94661252699862e2. Latest consensus state: current_term: 2 leader_uuid: "03bd6c71fc1d465e94661252699862e2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:58.551731 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:08:58.552163 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "03bd6c71fc1d465e94661252699862e2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:58.552251 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:58.552253 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "03bd6c71fc1d465e94661252699862e2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "9f19f0db7ee04f8c9ccbc5bf0bf1435c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37777 } } peers { permanent_uuid: "03bd6c71fc1d465e94661252699862e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "9b8e5dffeff44715a5730627d59b24bf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34651 } attrs { promote: false } } }
I20260924 03:08:58.552327 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8e5dffeff44715a5730627d59b24bf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:08:58.557011 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.553789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.0.1:43350)
0924 03:08:58.553887 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.553896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.553933 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.554525 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.554526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.554528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:58.554560 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.554561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555432 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.555433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556073 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.556075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556759 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556832 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556836 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.556901 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.556941 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":2}
I20260924 03:08:58.557027 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.554029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.98.1:46103)
0924 03:08:58.554388 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.554390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.554397 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:58.554422 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.554423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.554432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.554615 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:58.554650 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555364 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556164 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556680 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556752 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556765 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.556909 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.556959 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":79,"threads_started":1}
I20260924 03:08:58.557091 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.553849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.0.1:54812)
0924 03:08:58.553919 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.553930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.553976 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.554299 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.554300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.554302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:58.554345 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.554346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555091 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.555092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556163 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.556165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556827 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556890 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556894 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.556942 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.556985 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:08:58.557260 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.554047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.0.98.2:43695)
0924 03:08:58.554176 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.554192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.554199 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:08:58.554221 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.554221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.554230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.554316 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:58.554448 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555169 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556327 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556835 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556913 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.557121 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.557173 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 03:08:58.557314 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.553899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.0.98.3:46515)
0924 03:08:58.554162 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.554164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.554172 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:08:58.554221 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.554222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.554232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.554296 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:08:58.554397 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555010 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556229 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556717 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556821 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556833 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.557043 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.557251 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":50,"threads_started":1}
I20260924 03:08:58.557482 7666 tablet_service.cc:1467] Processing CreateTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 (DEFAULT_TABLE table=test-table [id=7f7a28e25e364c6d8f1242036b98fdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:58.557526 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 (DEFAULT_TABLE table=test-table [id=7f7a28e25e364c6d8f1242036b98fdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:58.557641 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c60485ace4a5da0c595f5272f9cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:58.557642 7666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c60485ace4a5da0c595f5272f9cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:58.558849 7825 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: Bootstrap starting.
I20260924 03:08:58.559049 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.553858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43695 (local address 127.0.0.1:37382)
0924 03:08:58.554020 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.554028 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.554085 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.554319 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:08:58.554320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:08:58.554321 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:08:58.554368 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.554369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.555265 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.555266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556247 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:08:58.556249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.556941 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.556942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:08:58.556995 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.556998 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:08:58.557045 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:08:58.557083 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":21,"thread_start_us":61,"threads_started":1}
I20260924 03:08:58.559188 7826 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Bootstrap starting.
I20260924 03:08:58.559268 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 (DEFAULT_TABLE table=test-table [id=7f7a28e25e364c6d8f1242036b98fdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:08:58.559377 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c60485ace4a5da0c595f5272f9cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:58.559412 7825 tablet_bootstrap.cc:654] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:58.559659 7826 tablet_bootstrap.cc:654] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:58.559971 7825 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: No bootstrap required, opened a new log
I20260924 03:08:58.560052 7825 ts_tablet_manager.cc:1397] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:58.560159 7826 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: No bootstrap required, opened a new log
I20260924 03:08:58.560216 7826 ts_tablet_manager.cc:1397] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:58.560238 7825 raft_consensus.cc:348] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.560312 7825 raft_consensus.cc:374] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:58.560338 7825 raft_consensus.cc:729] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6f6aa781794f29af39e98efe90e101, State: Initialized, Role: FOLLOWER
I20260924 03:08:58.560320 7826 raft_consensus.cc:348] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.560379 7826 raft_consensus.cc:374] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:58.560400 7826 raft_consensus.cc:729] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee73197bf4814bdf8180f1ace4605ccf, State: Initialized, Role: FOLLOWER
I20260924 03:08:58.560402 7825 consensus_queue.cc:260] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.560456 7826 consensus_queue.cc:260] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.560555 7825 ts_tablet_manager.cc:1428] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.560593 7826 ts_tablet_manager.cc:1428] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:08:58.561015 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.560791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:42791)
0924 03:08:58.560954 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.560992 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 03:08:58.561236 7734 heartbeater.cc:503] Master 127.0.98.61:42329 was elected leader, sending a full tablet report...
I20260924 03:08:58.561538 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.561414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:41405)
0924 03:08:58.561475 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.561510 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 03:08:58.561599 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.561345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:38985)
0924 03:08:58.561505 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.561565 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260924 03:08:58.561849 7462 heartbeater.cc:503] Master 127.0.98.61:42329 was elected leader, sending a full tablet report...
I20260924 03:08:58.562103 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.562006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:37871)
0924 03:08:58.562048 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.562069 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:08:58.562079 7831 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Bootstrap starting.
I20260924 03:08:58.562470 7831 tablet_bootstrap.cc:654] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:58.562942 7831 tablet_bootstrap.cc:492] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: No bootstrap required, opened a new log
I20260924 03:08:58.562988 7831 ts_tablet_manager.cc:1397] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:08:58.563087 7831 raft_consensus.cc:348] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.563156 7831 raft_consensus.cc:374] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:58.563169 7831 raft_consensus.cc:729] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad6f36e5aed4b09a938f7897a768ddb, State: Initialized, Role: FOLLOWER
I20260924 03:08:58.563223 7831 consensus_queue.cc:260] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.563377 7831 ts_tablet_manager.cc:1428] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:58.563727 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.563524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:47217)
0924 03:08:58.563668 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.563700 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260924 03:08:58.563812 7598 heartbeater.cc:503] Master 127.0.98.61:42329 was elected leader, sending a full tablet report...
I20260924 03:08:58.564157 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.563910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:47717)
0924 03:08:58.564108 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.564134 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260924 03:08:58.592613 7463 tablet.cc:2369] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:08:58.609273 7735 tablet.cc:2369] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:08:58.669240 7830 raft_consensus.cc:482] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:08:58.669318 7830 raft_consensus.cc:504] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.669472 7830 leader_election.cc:288] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103), 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
I20260924 03:08:58.669638 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c60485ace4a5da0c595f5272f9cc0" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d6f6aa781794f29af39e98efe90e101" is_pre_election: true
I20260924 03:08:58.669734 7414 raft_consensus.cc:2454] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 0.
I20260924 03:08:58.669647 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c60485ace4a5da0c595f5272f9cc0" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" is_pre_election: true
I20260924 03:08:58.669775 7550 raft_consensus.cc:2454] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 0.
I20260924 03:08:58.669899 7621 leader_election.cc:302] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6f6aa781794f29af39e98efe90e101, ee73197bf4814bdf8180f1ace4605ccf; no voters:
I20260924 03:08:58.670020 7830 raft_consensus.cc:2781] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:08:58.670060 7830 raft_consensus.cc:482] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:08:58.670080 7830 raft_consensus.cc:3037] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:58.670746 7830 raft_consensus.cc:504] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.670887 7830 leader_election.cc:288] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 election: Requested vote from peers 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103), 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695)
I20260924 03:08:58.671007 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c60485ace4a5da0c595f5272f9cc0" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d6f6aa781794f29af39e98efe90e101"
I20260924 03:08:58.671041 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c60485ace4a5da0c595f5272f9cc0" candidate_uuid: "ee73197bf4814bdf8180f1ace4605ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad6f36e5aed4b09a938f7897a768ddb"
I20260924 03:08:58.671082 7414 raft_consensus.cc:3037] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:58.671108 7550 raft_consensus.cc:3037] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:58.671787 7414 raft_consensus.cc:2454] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 1.
I20260924 03:08:58.671787 7550 raft_consensus.cc:2454] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73197bf4814bdf8180f1ace4605ccf in term 1.
I20260924 03:08:58.671981 7619 leader_election.cc:302] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad6f36e5aed4b09a938f7897a768ddb, ee73197bf4814bdf8180f1ace4605ccf; no voters:
I20260924 03:08:58.672079 7830 raft_consensus.cc:2781] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:58.672247 7830 raft_consensus.cc:686] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 1 LEADER]: Becoming Leader. State: Replica: ee73197bf4814bdf8180f1ace4605ccf, State: Running, Role: LEADER
I20260924 03:08:58.672312 7830 consensus_queue.cc:237] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } }
I20260924 03:08:58.672600 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.672482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:37159)
0924 03:08:58.672529 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.672568 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:08:58.672840 7093 catalog_manager.cc:5495] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ee73197bf4814bdf8180f1ace4605ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee73197bf4814bdf8180f1ace4605ccf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20260924 03:08:58.700778 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.698549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.98.3:46515)
0924 03:08:58.698593 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:58.698595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:58.698604 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:08:58.698659 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:58.698660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:58.698673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:58.698738 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:08:58.698863 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.699314 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.700113 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:58.700303 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:58.700384 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:58.700522 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:08:58.700558 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:08:58.700616 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:58.700675 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:08:58.700697 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:08:58.701550 7550 raft_consensus.cc:1261] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Refusing update from remote peer ee73197bf4814bdf8180f1ace4605ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:58.701586 7414 raft_consensus.cc:1261] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Refusing update from remote peer ee73197bf4814bdf8180f1ace4605ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:08:58.701772 7830 consensus_queue.cc:1035] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad6f36e5aed4b09a938f7897a768ddb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:58.701897 7837 consensus_queue.cc:1035] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6f6aa781794f29af39e98efe90e101" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:08:58.702651 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.702548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:52861)
0924 03:08:58.702593 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.702621 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:08:58.702816 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.702717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:58605)
0924 03:08:58.702759 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.702785 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:08:58.703717 7840 mvcc.cc:205] Tried to move back new op lower bound from 7332738411319803904 to 7332738411202220032. Current Snapshot: MvccSnapshot[applied={T|T < 7332738411319803904 or (T in {7332738411319803904})}]
I20260924 03:08:58.703934 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.703805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:60033)
0924 03:08:58.703850 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.703901 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:08:58.704257 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.704144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.1:42867)
0924 03:08:58.704178 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.704212 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 03:08:58.704563 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.704144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.3:54375)
0924 03:08:58.704506 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.704543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
I20260924 03:08:58.704640 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:58.704560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37777 (local address 127.0.98.2:56369)
0924 03:08:58.704593 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.704615 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 03:08:58.705919 7093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:58.705994 7093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:08:58.707177 7093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03bd6c71fc1d465e94661252699862e2: Sending DeleteTablet for 3 replicas of tablet 331c60485ace4a5da0c595f5272f9cc0
I20260924 03:08:58.707535 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37382
I20260924 03:08:58.707547 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43350
I20260924 03:08:58.707693 7666 tablet_service.cc:1514] Processing DeleteTablet for tablet 331c60485ace4a5da0c595f5272f9cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54812
I20260924 03:08:58.707752 7849 tablet_replica.cc:330] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: stopping tablet replica
I20260924 03:08:58.707818 7849 raft_consensus.cc:2229] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:58.707885 7849 raft_consensus.cc:2258] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:58.708105 7851 tablet_replica.cc:330] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: stopping tablet replica
I20260924 03:08:58.708163 7851 raft_consensus.cc:2229] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:08:58.708170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7331
I20260924 03:08:58.708226 7851 raft_consensus.cc:2258] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:58.707986 7850 tablet_replica.cc:330] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101: stopping tablet replica
I20260924 03:08:58.708392 7850 raft_consensus.cc:2229] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:08:58.708432 7850 raft_consensus.cc:2258] T 331c60485ace4a5da0c595f5272f9cc0 P 5d6f6aa781794f29af39e98efe90e101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:08:58.709280 7851 ts_tablet_manager.cc:1905] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:58.709895 7849 ts_tablet_manager.cc:1905] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:08:58.711061 7849 ts_tablet_manager.cc:1918] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:58.711100 7851 ts_tablet_manager.cc:1918] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:08:58.711130 7849 log.cc:1196] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/331c60485ace4a5da0c595f5272f9cc0
I20260924 03:08:58.711146 7851 log.cc:1196] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/331c60485ace4a5da0c595f5272f9cc0
I20260924 03:08:58.711347 7851 ts_tablet_manager.cc:1939] T 331c60485ace4a5da0c595f5272f9cc0 P ee73197bf4814bdf8180f1ace4605ccf: Deleting consensus metadata
I20260924 03:08:58.711352 7849 ts_tablet_manager.cc:1939] T 331c60485ace4a5da0c595f5272f9cc0 P 3ad6f36e5aed4b09a938f7897a768ddb: Deleting consensus metadata
I20260924 03:08:58.711691 7078 catalog_manager.cc:4841] TS 3ad6f36e5aed4b09a938f7897a768ddb (127.0.98.2:43695): tablet 331c60485ace4a5da0c595f5272f9cc0 (table test-table [id=7f7a28e25e364c6d8f1242036b98fdca]) successfully deleted
I20260924 03:08:58.713246 7081 catalog_manager.cc:4841] TS ee73197bf4814bdf8180f1ace4605ccf (127.0.98.3:46515): tablet 331c60485ace4a5da0c595f5272f9cc0 (table test-table [id=7f7a28e25e364c6d8f1242036b98fdca]) successfully deleted
W20260924 03:08:58.714319 7821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:08:58.713332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46103 (local address 127.0.0.1:43358)
0924 03:08:58.713385 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:08:58.713394 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:08:58.713433 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:08:58.714288 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 03:08:58.714435 7080 catalog_manager.cc:4584] TS 5d6f6aa781794f29af39e98efe90e101 (127.0.98.1:46103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 331c60485ace4a5da0c595f5272f9cc0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:08:58.714501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7467
I20260924 03:08:58.719796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7603
I20260924 03:08:58.724983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 6960
I20260924 03:08:58.729554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7148
2026-09-24T03:08:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4523 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:08:58 dist-test-slave-jtnz krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:08:58 dist-test-slave-jtnz krb5kdc[7854](info): set up 2 sockets
Sep 24 03:08:58 dist-test-slave-jtnz krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260924 03:08:59.730403 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:08:59 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:08:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 03:08:59.835031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:08:59.909777 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:08:59.909982 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:08:59.910004 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:08:59.911145 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:08:59.911185 7867 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:08:59.911197 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:08:59.911233 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:08:59.911260 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:08:59.911278 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:08:59.912638 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44459
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:08:59.912884 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:08:59.913111 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:08:59.917546 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:59.917634 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:08:59.917655 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:08:59.917893 7867 server_base.cc:1034] running on GCE node
I20260924 03:08:59.918130 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:08:59.918424 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:08:59.919564 7867 hybrid_clock.cc:648] HybridClock initialized: now 1790219339919546 us; error 35 us; skew 500 ppm
Sep 24 03:08:59 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:08:59.921741 7867 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 03:08:59.922259 7867 webserver.cc:460] Webserver started at http://127.0.98.62:36515/ using document root <none> and password file <none>
I20260924 03:08:59.922497 7867 fs_manager.cc:359] Metadata directory not provided
I20260924 03:08:59.922560 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:08:59.922696 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:08:59.923636 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "f6220237b55b4f279eebc5c26cee83fb"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-jtnz"
server_key: "ce4661036faf0030d244335aa989c444"
server_key_iv: "1869e9cc85a1c6b28b8b6ce56d189b36"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.923980 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6220237b55b4f279eebc5c26cee83fb"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-jtnz"
server_key: "ce4661036faf0030d244335aa989c444"
server_key_iv: "1869e9cc85a1c6b28b8b6ce56d189b36"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.925189 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:08:59.925804 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:08:59.925951 7867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:08:59.926066 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "f6220237b55b4f279eebc5c26cee83fb"
format_stamp: "Formatted at 2026-09-24 03:08:59 on dist-test-slave-jtnz"
server_key: "ce4661036faf0030d244335aa989c444"
server_key_iv: "1869e9cc85a1c6b28b8b6ce56d189b36"
server_key_version: "encryptionkey@0"
I20260924 03:08:59.926149 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:08:59.955147 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:08:59.956683 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:08:59.956820 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:08:59.962044 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44459
I20260924 03:08:59.962082 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44459 every 8 connection(s)
I20260924 03:08:59.962402 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:08:59.962944 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:08:59.964993 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Bootstrap starting.
I20260924 03:08:59.965534 7945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:08:59.965797 7945 log.cc:824] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:08:59.966392 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: No bootstrap required, opened a new log
I20260924 03:08:59.968340 7945 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 03:08:59.968490 7945 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:08:59.968534 7945 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6220237b55b4f279eebc5c26cee83fb, State: Initialized, Role: FOLLOWER
I20260924 03:08:59.968675 7945 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 03:08:59.968779 7945 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:08:59.968820 7945 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:08:59.968870 7945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:08:59.968890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7867
I20260924 03:08:59.968999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:08:59.969167 392 external_mini_cluster.cc:1428] Setting key e46c4b2945852a1af86e197083a3ee6e
I20260924 03:08:59.969462 7945 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 03:08:59.969790 7945 leader_election.cc:302] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6220237b55b4f279eebc5c26cee83fb; no voters:
I20260924 03:08:59.969969 7945 leader_election.cc:288] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:08:59.970003 7950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:08:59.970099 7950 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [term 1 LEADER]: Becoming Leader. State: Replica: f6220237b55b4f279eebc5c26cee83fb, State: Running, Role: LEADER
I20260924 03:08:59.970194 7950 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 03:08:59.970249 7945 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:08:59.970479 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6220237b55b4f279eebc5c26cee83fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260924 03:08:59.970535 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:59.970640 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6220237b55b4f279eebc5c26cee83fb. Latest consensus state: current_term: 1 leader_uuid: "f6220237b55b4f279eebc5c26cee83fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6220237b55b4f279eebc5c26cee83fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260924 03:08:59.970697 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb [sys.catalog]: This master's current role is: LEADER
I20260924 03:08:59.970865 7954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:08:59.971406 7954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:08:59.973672 7954 catalog_manager.cc:1347] Generated new cluster ID: 65d7a4fd42eb44d5851286bd14fe9f8b
I20260924 03:08:59.973722 7954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:08:59 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:08:59.979456 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:08:59.969611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.62:44459)
0924 03:08:59.969772 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 03:08:59.969778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:08:59.969800 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:08:59.969923 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:08:59.969929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:08:59.969985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:08:59.970164 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:08:59.970909 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.972267 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.973335 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:08:59.973784 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:08:59.973881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:08:59.975926 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:08:59.975930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:08:59.975930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:08:59.975940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:08:59.977970 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.978400 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.978403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.978403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.978486 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:08:59.978699 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:08:59.978700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:08:59.978700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:08:59.978898 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:08:59.979078 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:08:59.979175 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 03:08:59.979312 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20260924 03:08:59.980039 7954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:08:59.980656 7954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:08:59.985497 7954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Generated new TSK 0
I20260924 03:08:59.985647 7954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 03:09:00.013286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:00.107625 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:00.107823 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:00.107846 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:00.108939 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:00.108979 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:00.108999 7971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:00.109015 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:00.110450 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:00.110647 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:00.110863 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:00.116266 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:00.116282 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.116348 7971 server_base.cc:1034] running on GCE node
W20260924 03:09:00.116573 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.116789 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:00.117022 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:00.118216 7971 hybrid_clock.cc:648] HybridClock initialized: now 1790219340118199 us; error 30 us; skew 500 ppm
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:00.121641 7971 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:09:00.122231 7971 webserver.cc:460] Webserver started at http://127.0.98.1:38725/ using document root <none> and password file <none>
I20260924 03:09:00.122481 7971 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:00.122553 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:00.122692 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:00.124099 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "2eb1e41619b89be240728e6fc7a43e5f"
server_key_iv: "c74b8d193e7490379a3d8f8f39b6be9f"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.124521 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "2eb1e41619b89be240728e6fc7a43e5f"
server_key_iv: "c74b8d193e7490379a3d8f8f39b6be9f"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.126138 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 03:09:00.127378 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.129849 7971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:09:00.129948 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "2eb1e41619b89be240728e6fc7a43e5f"
server_key_iv: "c74b8d193e7490379a3d8f8f39b6be9f"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.130262 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:00.147388 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:00.150034 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:00.150159 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:00.150728 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:00.150765 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.150784 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:00.150821 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.157733 7971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42057
I20260924 03:09:00.157891 8101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42057 every 8 connection(s)
I20260924 03:09:00.158123 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:00.162222 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.151202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50721 (local address 127.0.98.62:44459)
0924 03:09:00.151242 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.151245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.151523 (+ 278us) server_negotiation.cc:407] Connection header received
0924 03:09:00.152200 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.152201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.152217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.152292 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.153216 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.154005 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.155662 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.155971 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.156033 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.159287 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.159289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.159289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.159298 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.160780 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.161247 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.161249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.161249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.161309 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.161561 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.161562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.161563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.161674 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.161744 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.162030 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.162111 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:09:00.162379 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.151209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.1:50721)
0924 03:09:00.151369 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.151415 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.152100 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.152415 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.152422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.152811 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.153096 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.153102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.154127 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.154127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.155550 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.155553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.156133 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.156135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.156277 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.157369 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.157377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.159213 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.160879 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.160886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.160887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.161131 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.161375 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.161376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.161376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.161492 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.161750 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.161756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.161891 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.161990 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.162132 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260924 03:09:00.162858 8102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 03:09:00.162935 8102 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:00.163111 8102 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 03:09:00.163614 7894 ts_manager.cc:194] Registered new tserver with Master: e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057)
I20260924 03:09:00.164382 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50721
I20260924 03:09:00.167872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 7971
I20260924 03:09:00.167961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:00.168145 392 external_mini_cluster.cc:1428] Setting key 049bce3c3392b1c86a58a445ed8e1475
I20260924 03:09:00.169646 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.163831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57587 (local address 127.0.98.62:44459)
0924 03:09:00.163865 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.163867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.163972 (+ 105us) server_negotiation.cc:407] Connection header received
0924 03:09:00.164040 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.164041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.164052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.164101 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.164543 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.165248 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.166310 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.166488 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.166543 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.167186 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.167187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.167187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.167193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.168515 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.168874 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.168875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.168875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.168928 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.169156 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.169156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.169157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.169261 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.169334 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.169493 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.169561 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:09:00.169663 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.163790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.1:57587)
0924 03:09:00.163911 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.163921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.163986 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.164142 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.164143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.164361 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.164453 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.164455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.165382 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.165382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.166229 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.166231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.166543 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.166544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.166585 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.166885 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.166888 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.167122 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.168585 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.168585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.168586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.168794 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.169031 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.169031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.169032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.169094 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.169327 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.169327 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.169367 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.169468 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.169561 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 03:09:00.192373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:00.270398 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:00.270586 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:00.270609 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:00.271735 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:00.271804 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:00.271840 8111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:00.271865 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:00.273274 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:00.273437 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:00.273597 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:00.278234 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.278306 8111 server_base.cc:1034] running on GCE node
W20260924 03:09:00.278223 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:00.278183 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.278654 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:00.278894 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:00.280051 8111 hybrid_clock.cc:648] HybridClock initialized: now 1790219340280039 us; error 27 us; skew 500 ppm
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:00.282414 8111 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:09:00.282809 8111 webserver.cc:460] Webserver started at http://127.0.98.2:33801/ using document root <none> and password file <none>
I20260924 03:09:00.283021 8111 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:00.283088 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:00.283236 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:00.284219 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "75815b3e01064b56810d64939382a32c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "42a665a98e3be0dbef2365f77f595c30"
server_key_iv: "dfc62ad4c34774cfa5bc0d247f4c39ba"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.284602 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "75815b3e01064b56810d64939382a32c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "42a665a98e3be0dbef2365f77f595c30"
server_key_iv: "dfc62ad4c34774cfa5bc0d247f4c39ba"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.285953 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:09:00.286705 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.286861 8111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.286955 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "75815b3e01064b56810d64939382a32c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "42a665a98e3be0dbef2365f77f595c30"
server_key_iv: "dfc62ad4c34774cfa5bc0d247f4c39ba"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.287045 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:00.301079 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:00.303243 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:00.303395 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:00.303838 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:00.303900 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.303937 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:00.303989 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:00.311185 8111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46855
I20260924 03:09:00.311270 8241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46855 every 8 connection(s)
I20260924 03:09:00.311544 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:00.313547 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.304310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34137 (local address 127.0.98.62:44459)
0924 03:09:00.304380 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.304382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.304627 (+ 245us) server_negotiation.cc:407] Connection header received
0924 03:09:00.305290 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.305291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.305305 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.305387 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.306279 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.306791 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.307808 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.308016 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.308100 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.310556 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.310557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.310558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.310565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.312241 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.312619 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.312621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.312621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.312677 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.312926 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.312927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.312927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.313046 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.313121 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.313375 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.313458 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:09:00.313711 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.304288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.2:34137)
0924 03:09:00.304479 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.304536 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.305174 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.305474 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.305478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.305852 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.306162 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.306168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.306895 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.306896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.307705 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.307707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.308109 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.308111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.308243 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.308723 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.308738 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.310455 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.312315 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.312320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.312322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.312530 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.312750 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.312751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.312752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.312857 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.313112 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.313117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.313248 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.313354 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.313498 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260924 03:09:00.314114 8242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 03:09:00.314210 8242 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:00.314384 8242 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 03:09:00.314740 7899 ts_manager.cc:194] Registered new tserver with Master: 75815b3e01064b56810d64939382a32c (127.0.98.2:46855)
I20260924 03:09:00.315179 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34137
I20260924 03:09:00.316308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8111
I20260924 03:09:00.316421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:00.316586 392 external_mini_cluster.cc:1428] Setting key 688c4f83a411caf1c5094fdd5573761a
I20260924 03:09:00.320499 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.315130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57251 (local address 127.0.98.62:44459)
0924 03:09:00.315166 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.315168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.315249 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:09:00.315303 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.315304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.315312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.315386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.315683 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.316119 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.317015 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.317288 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.317348 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.318208 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.318209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.318210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.318216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.319471 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.319848 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.319848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.319849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.319892 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.320084 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.320085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.320085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.320201 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.320248 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.320394 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.320442 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:09:00.320617 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.315055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.2:57251)
0924 03:09:00.315177 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.315187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.315245 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.315392 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.315392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.315555 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.315603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.315605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.316220 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.316221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.316908 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.316910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.317379 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.317380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.317442 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.317847 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.317852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.318107 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.319548 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.319549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.319549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.319774 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.319961 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.319962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.319963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.320028 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.320261 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.320262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.320307 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.320380 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.320481 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:09:00.339066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:00.412060 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:00.412279 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:00.412314 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:00.413450 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:00.413511 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:00.413544 8251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:09:00.413604 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:00.414997 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:00.415207 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:00.415422 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:00.419849 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.419852 8251 server_base.cc:1034] running on GCE node
W20260924 03:09:00.419822 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:00.419827 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:00.420221 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:00.420477 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:00.421623 8251 hybrid_clock.cc:648] HybridClock initialized: now 1790219340421615 us; error 23 us; skew 500 ppm
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:00.423779 8251 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:09:00.424077 8251 webserver.cc:460] Webserver started at http://127.0.98.3:41955/ using document root <none> and password file <none>
I20260924 03:09:00.424214 8251 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:00.424245 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:00.424314 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:00.425109 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "eb202fefa09d77b985926e71dd43a3bc"
server_key_iv: "e410800ef3ce8a3d80b6ddb8d214b523"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.425365 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "eb202fefa09d77b985926e71dd43a3bc"
server_key_iv: "e410800ef3ce8a3d80b6ddb8d214b523"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.426477 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:09:00.427074 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.427239 8251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:00.427317 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
format_stamp: "Formatted at 2026-09-24 03:09:00 on dist-test-slave-jtnz"
server_key: "eb202fefa09d77b985926e71dd43a3bc"
server_key_iv: "e410800ef3ce8a3d80b6ddb8d214b523"
server_key_version: "encryptionkey@0"
I20260924 03:09:00.427397 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:00.447973 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:00.449461 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:00.449601 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:00.450095 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:00.450134 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.450177 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:00.450238 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:00.456238 8251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40431
I20260924 03:09:00.456350 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40431 every 8 connection(s)
I20260924 03:09:00.456573 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:00.458973 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.450520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47277 (local address 127.0.98.62:44459)
0924 03:09:00.450584 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.450586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.450754 (+ 168us) server_negotiation.cc:407] Connection header received
0924 03:09:00.451200 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.451202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.451215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.451285 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.451978 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.452726 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.454125 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.454373 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.454445 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.456301 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.456303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.456303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.456310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.457625 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.458001 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.458002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.458003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.458057 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.458355 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.458356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.458356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.458472 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.458548 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.458798 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.458885 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:09:00.459097 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.450506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.3:47277)
0924 03:09:00.450647 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.450681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.451103 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.451337 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.451340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.451642 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.451877 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.451883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.452838 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.452839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.454043 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.454046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.454453 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.454454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.454571 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.455030 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.455039 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.456215 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.457712 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.457718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.457719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.457927 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.458145 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.458146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.458146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.458284 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.458538 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.458543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.458669 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.458763 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.458900 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260924 03:09:00.459537 8382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 03:09:00.459609 8382 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:00.459770 8382 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 03:09:00.460217 7899 ts_manager.cc:194] Registered new tserver with Master: c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:00.460651 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47277
I20260924 03:09:00.462888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8251
I20260924 03:09:00.462994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:00.463199 392 external_mini_cluster.cc:1428] Setting key c10a05c58ab75d93afb8445bf7698996
I20260924 03:09:00.464071 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:09:00.466086 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.460454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.3:58845)
0924 03:09:00.460572 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.460582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.460630 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.460909 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.460910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.461091 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:00.461135 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.461136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.461732 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.461732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.462310 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.462312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.462667 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.462668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.462729 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.463165 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:00.463169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:00.463444 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:00.465089 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.465090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.465090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.465312 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.465494 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:00.465494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:00.465495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:00.465564 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:00.465810 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:00.465811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:00.465857 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.465932 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.466022 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260924 03:09:00.466176 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.460746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58845 (local address 127.0.98.62:44459)
0924 03:09:00.460799 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.460801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.460808 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:00.460825 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.460826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.460834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.460888 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.461221 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.461639 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.462397 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.462589 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.462665 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.463538 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.463539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.463539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.463545 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.465011 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.465388 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.465389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.465389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.465433 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.465622 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.465623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.465623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.465709 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.465808 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.465967 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.466049 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:09:00.484081 7949 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:00.479297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.0.98.62:44459)
0924 03:09:00.479382 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.479397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.479407 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:00.479435 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.479437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.479446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.479531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.480002 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.480431 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.481105 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.481268 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.481314 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.483982 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34568: BlockingRecv error: recv got EOF from 127.0.0.1:34568 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 03:09:00 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:00.490638 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.484699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.62:44459)
0924 03:09:00.484750 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.484751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.484845 (+ 94us) server_negotiation.cc:407] Connection header received
0924 03:09:00.484878 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.484878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.484885 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.484929 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:00.485317 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.485904 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.486544 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.486702 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.486749 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.488506 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:00.488507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:00.488507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:00.488513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:00.489669 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.489985 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.489986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.489986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.490032 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:00.490232 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:00.490233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:00.490233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:00.490323 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:00.490393 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.490489 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.490538 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:09:00.491896 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:00.493358 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:00.499024 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46855 (local address 127.0.0.1:45792)
0924 03:09:00.495704 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.495712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.495772 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.495942 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.495953 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.495955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.496023 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.496025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.497268 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.497269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.497913 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.497914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.498785 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.498842 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.498845 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.498873 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.498929 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260924 03:09:00.499049 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.0.98.1:42057)
0924 03:09:00.495913 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.495917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.495926 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:00.495957 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.495959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.495972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.496034 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.496192 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.496860 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498117 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.498650 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498727 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.498800 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.498909 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.498944 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260924 03:09:00.499011 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42057 (local address 127.0.0.1:38276)
0924 03:09:00.495767 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.495778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.495827 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.496045 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.496046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.496047 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.496093 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.496095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.496942 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.496942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.498032 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.498034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.498739 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.498813 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.498826 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.498891 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.498930 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260924 03:09:00.499297 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.98.2:46855)
0924 03:09:00.495635 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.495639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.495773 (+ 134us) server_negotiation.cc:407] Connection header received
0924 03:09:00.495838 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.495840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.495857 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.495931 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.496151 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.497167 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498025 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.498695 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.499061 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.499134 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.499162 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.499203 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260924 03:09:00.499492 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.0.1:37132)
0924 03:09:00.495480 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.495514 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.495571 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.496120 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.496121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.496123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.496156 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.496157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.497246 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.497247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.498239 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.498241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.499261 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.499262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.499321 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.499324 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.499375 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.499408 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":22,"thread_start_us":60,"threads_started":1}
I20260924 03:09:00.499701 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.495554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.98.3:40431)
0924 03:09:00.495931 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.495935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.495944 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:00.495971 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.495973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.495986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.496060 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.496224 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.497173 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498307 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.498940 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.498998 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.499061 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.499582 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.499618 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":74,"threads_started":1}
I20260924 03:09:00.499818 8036 tablet_service.cc:1467] Processing CreateTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (DEFAULT_TABLE table=test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.499807 8176 tablet_service.cc:1467] Processing CreateTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (DEFAULT_TABLE table=test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.500204 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.500204 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.500425 8316 tablet_service.cc:1467] Processing CreateTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (DEFAULT_TABLE table=test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.500733 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.503535 8410 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Bootstrap starting.
I20260924 03:09:00.503847 8411 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Bootstrap starting.
I20260924 03:09:00.504172 8410 tablet_bootstrap.cc:654] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.504401 8411 tablet_bootstrap.cc:654] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.504482 8410 log.cc:824] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:00.504714 8411 log.cc:824] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:00.504824 8412 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Bootstrap starting.
I20260924 03:09:00.505211 8410 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: No bootstrap required, opened a new log
I20260924 03:09:00.505283 8410 ts_tablet_manager.cc:1397] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:00.505332 8412 tablet_bootstrap.cc:654] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.505396 8411 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: No bootstrap required, opened a new log
I20260924 03:09:00.505458 8411 ts_tablet_manager.cc:1397] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:00.505622 8412 log.cc:824] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:00.506290 8412 tablet_bootstrap.cc:492] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: No bootstrap required, opened a new log
I20260924 03:09:00.506350 8412 ts_tablet_manager.cc:1397] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:00.506940 8410 raft_consensus.cc:348] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.507076 8410 raft_consensus.cc:374] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.507112 8410 raft_consensus.cc:729] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0164b2a35d34e6d96e78a8b77566a9c, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.507207 8410 consensus_queue.cc:260] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.507192 8411 raft_consensus.cc:348] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.507344 8411 raft_consensus.cc:374] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.507373 8411 raft_consensus.cc:729] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.507465 8410 ts_tablet_manager.cc:1428] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:00.507483 8411 consensus_queue.cc:260] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.507575 8382 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260924 03:09:00.507725 8102 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260924 03:09:00.507711 8411 ts_tablet_manager.cc:1428] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:09:00.508128 8412 raft_consensus.cc:348] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.508270 8412 raft_consensus.cc:374] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.508304 8412 raft_consensus.cc:729] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75815b3e01064b56810d64939382a32c, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.508452 8412 consensus_queue.cc:260] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.508687 8412 ts_tablet_manager.cc:1428] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:00.508751 8242 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
W20260924 03:09:00.562093 8243 tablet.cc:2369] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:00.658970 8103 tablet.cc:2369] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:09:00.707068 8383 tablet.cc:2369] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:00.864378 8418 raft_consensus.cc:482] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:00.864485 8418 raft_consensus.cc:504] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.864879 8418 leader_election.cc:288] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431), e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057)
I20260924 03:09:00.868500 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.865107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54027 (local address 127.0.98.3:40431)
0924 03:09:00.865157 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.865159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.865250 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:09:00.865304 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.865305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.865328 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.865378 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.865571 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.866268 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.867639 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.868148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.868225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.868234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.868398 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.868432 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:09:00.868501 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.865120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50419 (local address 127.0.98.1:42057)
0924 03:09:00.865170 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.865172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.865250 (+ 78us) server_negotiation.cc:407] Connection header received
0924 03:09:00.865304 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.865305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.865328 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.865378 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:00.865571 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.866268 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.867639 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.868148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.868225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.868234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.868399 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.868432 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:09:00.868536 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.865042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42057 (local address 127.0.98.2:50419)
0924 03:09:00.865175 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.865183 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.865243 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.865398 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.865398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.865402 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.865505 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.865506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.866341 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.866342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.867558 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.867561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.868234 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.868235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.868319 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.868326 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.868380 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.868438 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":126,"thread_start_us":68,"threads_started":1}
I20260924 03:09:00.868839 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f01b79a0cb4ecf94ee2ff06ab15e99" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" is_pre_election: true
I20260924 03:09:00.868541 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.865032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.98.2:54027)
0924 03:09:00.865166 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:00.865177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:00.865243 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:00.865398 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:00.865398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:00.865400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:00.865501 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.865503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.866341 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.866341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.867558 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:00.867561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.868232 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.868234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:00.868319 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.868325 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:09:00.868380 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:09:00.868438 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":60,"thread_start_us":54,"threads_started":1}
I20260924 03:09:00.868988 8056 raft_consensus.cc:2454] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:00.869216 8128 leader_election.cc:302] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:00.869237 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f01b79a0cb4ecf94ee2ff06ab15e99" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" is_pre_election: true
I20260924 03:09:00.869381 8336 raft_consensus.cc:2454] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:00.869365 8418 raft_consensus.cc:2781] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:00.869422 8418 raft_consensus.cc:482] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:00.869444 8418 raft_consensus.cc:3037] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.870301 8418 raft_consensus.cc:504] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.870432 8418 leader_election.cc:288] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Requested vote from peers c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431), e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057)
I20260924 03:09:00.870559 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f01b79a0cb4ecf94ee2ff06ab15e99" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
I20260924 03:09:00.870597 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f01b79a0cb4ecf94ee2ff06ab15e99" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
I20260924 03:09:00.870661 8336 raft_consensus.cc:3037] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.870673 8056 raft_consensus.cc:3037] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:00.871421 8056 raft_consensus.cc:2454] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:00.871433 8336 raft_consensus.cc:2454] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:00.871575 8128 leader_election.cc:302] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:00.871651 8418 raft_consensus.cc:2781] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:00.871762 8418 raft_consensus.cc:686] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 1 LEADER]: Becoming Leader. State: Replica: 75815b3e01064b56810d64939382a32c, State: Running, Role: LEADER
I20260924 03:09:00.871862 8418 consensus_queue.cc:237] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } }
I20260924 03:09:00.872594 7899 catalog_manager.cc:5495] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c reported cstate change: term changed from 0 to 1, leader changed from <none> to 75815b3e01064b56810d64939382a32c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:00.939280 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:00.937170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.0.98.2:46855)
0924 03:09:00.937220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:00.937221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:00.937228 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:00.937325 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:00.937326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:00.937341 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:00.937397 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:00.937528 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.937952 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.938664 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:00.938818 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:00.938862 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:00.938970 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:00.939049 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:00.939117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:00.939155 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:09:00.939170 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:09:00.940930 8336 raft_consensus.cc:1261] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.940930 8056 raft_consensus.cc:1261] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:00.941303 8418 consensus_queue.cc:1035] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.941429 8421 consensus_queue.cc:1035] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:00.945447 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.945533 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:00.946223 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Sending DeleteTablet for 3 replicas of tablet c1f01b79a0cb4ecf94ee2ff06ab15e99
I20260924 03:09:00.946533 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37132
I20260924 03:09:00.946571 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260924 03:09:00.946570 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45792
I20260924 03:09:00.946774 8434 tablet_replica.cc:330] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: stopping tablet replica
I20260924 03:09:00.946866 8434 raft_consensus.cc:2229] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.946961 8436 tablet_replica.cc:330] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: stopping tablet replica
I20260924 03:09:00.947050 8436 raft_consensus.cc:2229] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:00.947032 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:09:00.947130 8436 raft_consensus.cc:2258] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:09:00.947153 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:00.947273 8435 tablet_replica.cc:330] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: stopping tablet replica
I20260924 03:09:00.947346 8435 raft_consensus.cc:2229] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:00.947420 8435 raft_consensus.cc:2258] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.947770 8435 ts_tablet_manager.cc:1905] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.948277 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6e5dd43d764845a341a84c24ada0db (DEFAULT_TABLE table=test-table [id=dc6855e350b84a108a566096a7289bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.948297 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6e5dd43d764845a341a84c24ada0db (DEFAULT_TABLE table=test-table [id=dc6855e350b84a108a566096a7289bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.948400 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6e5dd43d764845a341a84c24ada0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.948403 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6e5dd43d764845a341a84c24ada0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.949388 8412 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Bootstrap starting.
I20260924 03:09:00.949651 8436 ts_tablet_manager.cc:1905] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.949786 8411 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Bootstrap starting.
I20260924 03:09:00.949779 8435 ts_tablet_manager.cc:1918] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.949793 8412 tablet_bootstrap.cc:654] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.949837 8435 log.cc:1196] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/c1f01b79a0cb4ecf94ee2ff06ab15e99
I20260924 03:09:00.950165 8411 tablet_bootstrap.cc:654] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.950171 8435 ts_tablet_manager.cc:1939] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting consensus metadata
I20260924 03:09:00.950384 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 0e6e5dd43d764845a341a84c24ada0db (DEFAULT_TABLE table=test-table [id=dc6855e350b84a108a566096a7289bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:00.950498 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e6e5dd43d764845a341a84c24ada0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:00.950563 7885 catalog_manager.cc:4841] TS c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431): tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (table test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]) successfully deleted
I20260924 03:09:00.951126 8436 ts_tablet_manager.cc:1918] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.951231 8436 log.cc:1196] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/c1f01b79a0cb4ecf94ee2ff06ab15e99
I20260924 03:09:00.951272 8410 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Bootstrap starting.
I20260924 03:09:00.951555 8436 ts_tablet_manager.cc:1939] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P 75815b3e01064b56810d64939382a32c: Deleting consensus metadata
I20260924 03:09:00.951659 8410 tablet_bootstrap.cc:654] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:00.951856 8434 raft_consensus.cc:2258] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:00.951973 7884 catalog_manager.cc:4841] TS 75815b3e01064b56810d64939382a32c (127.0.98.2:46855): tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (table test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]) successfully deleted
I20260924 03:09:00.952246 8410 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: No bootstrap required, opened a new log
I20260924 03:09:00.952293 8410 ts_tablet_manager.cc:1397] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:09:00.952299 8434 ts_tablet_manager.cc:1905] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:00.952416 8410 raft_consensus.cc:348] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.952471 8410 raft_consensus.cc:374] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.952489 8410 raft_consensus.cc:729] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0164b2a35d34e6d96e78a8b77566a9c, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.952533 8410 consensus_queue.cc:260] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.952605 8410 ts_tablet_manager.cc:1428] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:00.953544 8434 ts_tablet_manager.cc:1918] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:00.953603 8434 log.cc:1196] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/c1f01b79a0cb4ecf94ee2ff06ab15e99
I20260924 03:09:00.953789 8434 ts_tablet_manager.cc:1939] T c1f01b79a0cb4ecf94ee2ff06ab15e99 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting consensus metadata
I20260924 03:09:00.954037 8411 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: No bootstrap required, opened a new log
I20260924 03:09:00.954107 8411 ts_tablet_manager.cc:1397] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 03:09:00.954193 7884 catalog_manager.cc:4841] TS e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057): tablet c1f01b79a0cb4ecf94ee2ff06ab15e99 (table test-table [id=4f37d4fbe45d4f9ea4224ac0d9ed091f]) successfully deleted
I20260924 03:09:00.954264 8411 raft_consensus.cc:348] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.954334 8411 raft_consensus.cc:374] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.954357 8411 raft_consensus.cc:729] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.954383 8412 tablet_bootstrap.cc:492] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: No bootstrap required, opened a new log
I20260924 03:09:00.954396 8411 consensus_queue.cc:260] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.954425 8412 ts_tablet_manager.cc:1397] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 03:09:00.954500 8411 ts_tablet_manager.cc:1428] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:00.954550 8412 raft_consensus.cc:348] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.954607 8412 raft_consensus.cc:374] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:00.954632 8412 raft_consensus.cc:729] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75815b3e01064b56810d64939382a32c, State: Initialized, Role: FOLLOWER
I20260924 03:09:00.954674 8412 consensus_queue.cc:260] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:00.954762 8412 ts_tablet_manager.cc:1428] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.237759 8417 raft_consensus.cc:482] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:01.237841 8417 raft_consensus.cc:504] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.238173 8417 leader_election.cc:288] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75815b3e01064b56810d64939382a32c (127.0.98.2:46855), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.241977 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.238393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54647 (local address 127.0.98.2:46855)
0924 03:09:01.238438 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.238441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.238517 (+ 76us) server_negotiation.cc:407] Connection header received
0924 03:09:01.238556 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.238557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.238565 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.238630 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:01.238800 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.239466 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241053 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.241575 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241616 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.241626 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.241850 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.241912 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:09:01.241984 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.238411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57991 (local address 127.0.98.3:40431)
0924 03:09:01.238446 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.238448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.238518 (+ 70us) server_negotiation.cc:407] Connection header received
0924 03:09:01.238562 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.238562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.238570 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.238630 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:01.238799 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.239466 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241053 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.241574 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241616 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.241626 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.241851 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.241913 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:09:01.242002 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.238330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.98.1:57991)
0924 03:09:01.238438 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.238456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.238508 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.238646 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.238647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.238649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:01.238726 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.238728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.239561 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.239562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.240943 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.240946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.241697 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.241764 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.241768 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.241819 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.241873 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":84,"thread_start_us":54,"threads_started":1}
I20260924 03:09:01.242002 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.238330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46855 (local address 127.0.98.1:54647)
0924 03:09:01.238438 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.238447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.238508 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.238646 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.238647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.238650 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:01.238726 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.238729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.239541 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.239542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.240951 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.240953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.241703 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.241703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.241766 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.241769 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.241819 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.241873 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":75,"thread_start_us":44,"threads_started":1}
I20260924 03:09:01.242254 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6e5dd43d764845a341a84c24ada0db" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" is_pre_election: true
I20260924 03:09:01.242331 8336 raft_consensus.cc:2454] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 0.
I20260924 03:09:01.242479 7989 leader_election.cc:302] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0164b2a35d34e6d96e78a8b77566a9c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:01.242517 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6e5dd43d764845a341a84c24ada0db" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75815b3e01064b56810d64939382a32c" is_pre_election: true
I20260924 03:09:01.242604 8417 raft_consensus.cc:2781] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:01.242640 8417 raft_consensus.cc:482] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:01.242681 8417 raft_consensus.cc:3037] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.242664 8196 raft_consensus.cc:2454] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 0.
I20260924 03:09:01.243356 8417 raft_consensus.cc:504] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.243487 8417 leader_election.cc:288] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 election: Requested vote from peers 75815b3e01064b56810d64939382a32c (127.0.98.2:46855), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.243621 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6e5dd43d764845a341a84c24ada0db" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75815b3e01064b56810d64939382a32c"
I20260924 03:09:01.243700 8196 raft_consensus.cc:3037] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.243646 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e6e5dd43d764845a341a84c24ada0db" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
I20260924 03:09:01.243729 8336 raft_consensus.cc:3037] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.244267 8196 raft_consensus.cc:2454] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 1.
I20260924 03:09:01.244275 8336 raft_consensus.cc:2454] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 1.
I20260924 03:09:01.244436 7988 leader_election.cc:302] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:01.244521 8417 raft_consensus.cc:2781] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:01.244637 8417 raft_consensus.cc:686] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 1 LEADER]: Becoming Leader. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Running, Role: LEADER
I20260924 03:09:01.244693 8417 consensus_queue.cc:237] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.245328 7894 catalog_manager.cc:5495] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d reported cstate change: term changed from 0 to 1, leader changed from <none> to e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:09:01 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219339, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:01.324939 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.318852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.0.98.62:44459)
0924 03:09:01.318992 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.318994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.319009 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:09:01.319051 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.319052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.319073 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.319135 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:01.319506 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.319965 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.320577 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.320746 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.320792 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.322724 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:01.322726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:01.322726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:01.322731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:01.324030 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:01.324365 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:01.324366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:01.324366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:01.324416 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:01.324572 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:01.324572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:01.324572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:01.324661 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:01.324707 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.324811 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.324879 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":114,"thread_start_us":97,"threads_started":1}
I20260924 03:09:01.325094 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.318807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.0.1:34584)
0924 03:09:01.318945 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.318962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.319009 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.319148 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.319148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.319407 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:01.319446 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.319448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.320025 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.320025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.320517 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.320518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.320802 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.320802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.320837 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.321298 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:01.321302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:01.322653 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:01.324094 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:01.324100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:01.324102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:01.324305 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:01.324472 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:01.324472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:01.324472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:01.324522 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:01.324709 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:01.324714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:01.324750 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.324798 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.324940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":60,"thread_start_us":86,"threads_started":1}
I20260924 03:09:01.326805 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:09:01.326969 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:01.328323 8036 tablet_service.cc:1467] Processing CreateTablet for tablet f58eed1d5bf0495595bd57221804d53d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e020d17e6824ca49b6dcf6d00090a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:01.328332 8176 tablet_service.cc:1467] Processing CreateTablet for tablet f58eed1d5bf0495595bd57221804d53d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e020d17e6824ca49b6dcf6d00090a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:01.328361 8316 tablet_service.cc:1467] Processing CreateTablet for tablet f58eed1d5bf0495595bd57221804d53d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e020d17e6824ca49b6dcf6d00090a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:09:01.328469 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58eed1d5bf0495595bd57221804d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.328473 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58eed1d5bf0495595bd57221804d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.328487 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58eed1d5bf0495595bd57221804d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.329953 8410 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: Bootstrap starting.
I20260924 03:09:01.330066 8412 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: Bootstrap starting.
I20260924 03:09:01.330396 8412 tablet_bootstrap.cc:654] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.330396 8410 tablet_bootstrap.cc:654] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.330745 8411 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: Bootstrap starting.
I20260924 03:09:01.331001 8412 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: No bootstrap required, opened a new log
I20260924 03:09:01.331089 8412 ts_tablet_manager.cc:1397] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:09:01.331184 8411 tablet_bootstrap.cc:654] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.331218 8412 raft_consensus.cc:348] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.331291 8412 raft_consensus.cc:374] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.331314 8412 raft_consensus.cc:729] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75815b3e01064b56810d64939382a32c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.331367 8412 consensus_queue.cc:260] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.331465 8412 ts_tablet_manager.cc:1428] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.331516 8431 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.331616 8431 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.331920 8411 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: No bootstrap required, opened a new log
I20260924 03:09:01.331986 8411 ts_tablet_manager.cc:1397] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:01.332031 8410 tablet_bootstrap.cc:492] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: No bootstrap required, opened a new log
I20260924 03:09:01.332084 8410 ts_tablet_manager.cc:1397] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:01.332142 8411 raft_consensus.cc:348] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.332222 8411 raft_consensus.cc:374] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.332252 8411 raft_consensus.cc:729] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.332217 8410 raft_consensus.cc:348] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.332278 8410 raft_consensus.cc:374] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.332309 8410 raft_consensus.cc:729] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0164b2a35d34e6d96e78a8b77566a9c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.332299 8411 consensus_queue.cc:260] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.332368 8410 consensus_queue.cc:260] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.332401 8411 ts_tablet_manager.cc:1428] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.332468 8410 ts_tablet_manager.cc:1428] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.332446 8417 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.332538 8417 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.332552 8416 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.332633 8416 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.357690 8431 raft_consensus.cc:482] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:01.357739 8431 raft_consensus.cc:504] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.357870 8431 leader_election.cc:288] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.358045 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58eed1d5bf0495595bd57221804d53d" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" is_pre_election: true
I20260924 03:09:01.358047 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58eed1d5bf0495595bd57221804d53d" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" is_pre_election: true
I20260924 03:09:01.358119 8056 raft_consensus.cc:2454] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:01.358119 8336 raft_consensus.cc:2454] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:01.358245 8129 leader_election.cc:302] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, c0164b2a35d34e6d96e78a8b77566a9c; no voters:
I20260924 03:09:01.358348 8431 raft_consensus.cc:2781] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:01.358397 8431 raft_consensus.cc:482] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:01.358410 8431 raft_consensus.cc:3037] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.359217 8431 raft_consensus.cc:504] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.359373 8431 leader_election.cc:288] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Requested vote from peers e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.359519 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58eed1d5bf0495595bd57221804d53d" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
I20260924 03:09:01.359526 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58eed1d5bf0495595bd57221804d53d" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
I20260924 03:09:01.359596 8056 raft_consensus.cc:3037] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.359596 8336 raft_consensus.cc:3037] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.360169 8056 raft_consensus.cc:2454] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:01.360234 8336 raft_consensus.cc:2454] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:01.360358 8128 leader_election.cc:302] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:01.360436 8431 raft_consensus.cc:2781] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:01.360514 8431 raft_consensus.cc:686] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [term 1 LEADER]: Becoming Leader. State: Replica: 75815b3e01064b56810d64939382a32c, State: Running, Role: LEADER
I20260924 03:09:01.360570 8431 consensus_queue.cc:237] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.360831 8418 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: TxnStatusTablet state changed. Reason: New leader 75815b3e01064b56810d64939382a32c. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.360936 8418 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:01.361122 8455 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:09:01.361232 8455 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:09:01.361258 7894 catalog_manager.cc:5495] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c reported cstate change: term changed from 0 to 1, leader changed from <none> to 75815b3e01064b56810d64939382a32c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:01.369009 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.366782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.0.98.2:46855)
0924 03:09:01.366823 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.366825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.366834 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:01.366891 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.366892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.366901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.366953 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:01.367033 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.367657 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.368294 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.368537 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.368599 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.368772 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.368801 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.368848 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.368931 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.368951 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:09:01.369026 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.366713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46855 (local address 127.0.0.1:45822)
0924 03:09:01.366782 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.366793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.366837 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.366963 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.366964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.366965 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:01.366996 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.366998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.367721 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.367721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.368210 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.368212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.368601 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.368602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.368667 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.368684 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.368863 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.368870 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.368925 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.368965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:09:01.369773 8336 raft_consensus.cc:1261] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.369784 8056 raft_consensus.cc:1261] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.369993 8418 consensus_queue.cc:1035] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.370115 8418 consensus_queue.cc:1035] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.370607 8417 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: TxnStatusTablet state changed. Reason: New leader 75815b3e01064b56810d64939382a32c. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.370628 8416 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: TxnStatusTablet state changed. Reason: New leader 75815b3e01064b56810d64939382a32c. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.370683 8417 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.370689 8416 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.371354 8421 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.371423 8421 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:01.371548 8424 mvcc.cc:205] Tried to move back new op lower bound from 7332738422249533440 to 7332738422213427200. Current Snapshot: MvccSnapshot[applied={T|T < 7332738422249533440 or (T in {7332738422249533440})}]
I20260924 03:09:01.371601 8417 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.371671 8417 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P e08a61f20cbc4e5e9c8460a92bae271d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.375471 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.372952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.1:42057)
0924 03:09:01.373097 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.373099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.373106 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:01.373149 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.373149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.373166 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.373237 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:01.373310 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.373951 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.374677 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.374964 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.375053 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.375177 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.375279 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.375343 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.375381 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.375398 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260924 03:09:01.379925 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.377495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60127 (local address 127.0.98.2:46855)
0924 03:09:01.377549 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.377552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.377656 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:09:01.377726 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.377727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.377739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.377788 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:01.378211 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.378666 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.379339 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.379503 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.379564 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.379718 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.379745 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.379802 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.379853 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.379869 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:09:01.379999 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.377414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46855 (local address 127.0.98.1:60127)
0924 03:09:01.377579 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.377588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.377676 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.377793 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.377794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.377795 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:01.378119 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.378121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.378736 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.378737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.379251 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.379252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.379565 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.379565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.379626 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.379637 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.379804 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.379806 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.379843 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.379886 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260924 03:09:01.381724 8335 raft_consensus.cc:1261] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Refusing update from remote peer e08a61f20cbc4e5e9c8460a92bae271d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.381760 8196 raft_consensus.cc:1261] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Refusing update from remote peer e08a61f20cbc4e5e9c8460a92bae271d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.382001 8442 consensus_queue.cc:1035] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.382141 8417 consensus_queue.cc:1035] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.382647 8431 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.382726 8431 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P 75815b3e01064b56810d64939382a32c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:09:01.383463 8416 tablet_replica.cc:439] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260924 03:09:01.383534 8416 tablet_replica.cc:442] T f58eed1d5bf0495595bd57221804d53d P c0164b2a35d34e6d96e78a8b77566a9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:09:01.383889 8425 mvcc.cc:205] Tried to move back new op lower bound from 7332738422297714688 to 7332738421738782720. Current Snapshot: MvccSnapshot[applied={T|T < 7332738422297714688}]
I20260924 03:09:01.384023 8426 mvcc.cc:205] Tried to move back new op lower bound from 7332738422297714688 to 7332738421738782720. Current Snapshot: MvccSnapshot[applied={T|T < 7332738422297714688}]
I20260924 03:09:01.390278 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.387695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42057 (local address 127.0.98.2:59389)
0924 03:09:01.387841 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:01.387850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:01.387903 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:01.388018 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:01.388018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:01.388020 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:09:01.388056 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.388057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.388796 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.388796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.389421 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:01.389423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.389820 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.389820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:01.389867 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.389877 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.390099 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.390101 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:09:01.390144 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:09:01.390207 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260924 03:09:01.390290 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.387746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59389 (local address 127.0.98.1:42057)
0924 03:09:01.387776 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.387778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.387905 (+ 127us) server_negotiation.cc:407] Connection header received
0924 03:09:01.387948 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.387949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.387956 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.388017 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:01.388103 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.388709 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.389493 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.389753 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.389816 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.389984 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.390027 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.390073 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.390155 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.390213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:09:01.403975 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:01.404066 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:01.404630 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Sending DeleteTablet for 3 replicas of tablet 0e6e5dd43d764845a341a84c24ada0db
I20260924 03:09:01.404868 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6e5dd43d764845a341a84c24ada0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37132
I20260924 03:09:01.404891 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6e5dd43d764845a341a84c24ada0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260924 03:09:01.404944 8435 tablet_replica.cc:330] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: stopping tablet replica
I20260924 03:09:01.404970 8434 tablet_replica.cc:330] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: stopping tablet replica
I20260924 03:09:01.404983 8435 raft_consensus.cc:2229] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:01.405014 8435 raft_consensus.cc:2258] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.405014 8434 raft_consensus.cc:2229] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:01.405085 8434 raft_consensus.cc:2258] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.405298 8435 ts_tablet_manager.cc:1905] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.406236 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e6e5dd43d764845a341a84c24ada0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45792
I20260924 03:09:01.406311 8436 tablet_replica.cc:330] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: stopping tablet replica
I20260924 03:09:01.406351 8436 raft_consensus.cc:2229] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:01.406383 8436 raft_consensus.cc:2258] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.406595 8434 ts_tablet_manager.cc:1905] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.406734 8435 ts_tablet_manager.cc:1918] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:01.406806 8435 log.cc:1196] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/0e6e5dd43d764845a341a84c24ada0db
I20260924 03:09:01.406996 8435 ts_tablet_manager.cc:1939] T 0e6e5dd43d764845a341a84c24ada0db P c0164b2a35d34e6d96e78a8b77566a9c: Deleting consensus metadata
I20260924 03:09:01.407434 7885 catalog_manager.cc:4841] TS c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431): tablet 0e6e5dd43d764845a341a84c24ada0db (table test-table [id=dc6855e350b84a108a566096a7289bbe]) successfully deleted
I20260924 03:09:01.408012 8434 ts_tablet_manager.cc:1918] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:01.408070 8434 log.cc:1196] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/0e6e5dd43d764845a341a84c24ada0db
I20260924 03:09:01.408267 8434 ts_tablet_manager.cc:1939] T 0e6e5dd43d764845a341a84c24ada0db P e08a61f20cbc4e5e9c8460a92bae271d: Deleting consensus metadata
I20260924 03:09:01.408560 7884 catalog_manager.cc:4841] TS e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057): tablet 0e6e5dd43d764845a341a84c24ada0db (table test-table [id=dc6855e350b84a108a566096a7289bbe]) successfully deleted
I20260924 03:09:01.411217 8436 ts_tablet_manager.cc:1905] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.412606 8436 ts_tablet_manager.cc:1918] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:01.412652 8436 log.cc:1196] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/0e6e5dd43d764845a341a84c24ada0db
I20260924 03:09:01.412859 8436 ts_tablet_manager.cc:1939] T 0e6e5dd43d764845a341a84c24ada0db P 75815b3e01064b56810d64939382a32c: Deleting consensus metadata
I20260924 03:09:01.413196 7884 catalog_manager.cc:4841] TS 75815b3e01064b56810d64939382a32c (127.0.98.2:46855): tablet 0e6e5dd43d764845a341a84c24ada0db (table test-table [id=dc6855e350b84a108a566096a7289bbe]) successfully deleted
Sep 24 03:09:01 dist-test-slave-jtnz krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:09:01 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:01 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:09:01.434317 8450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:01.428035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.62:44459)
0924 03:09:01.428073 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.428075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.428202 (+ 127us) server_negotiation.cc:407] Connection header received
0924 03:09:01.428234 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.428235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.428245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.428314 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:01.428884 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.429597 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.430766 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.431101 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.431172 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.434265 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34588: BlockingRecv error: recv got EOF from 127.0.0.1:34588 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 03:09:01 dist-test-slave-jtnz krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:09:01.446609 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.435457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.0.98.62:44459)
0924 03:09:01.435487 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.435489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.438440 (+ 2951us) server_negotiation.cc:407] Connection header received
0924 03:09:01.438529 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.438530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.438539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.438593 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:01.439099 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.439747 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.440709 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.441025 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.441103 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.443638 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:01.443640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:01.443640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:01.443669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:01.445270 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:01.445657 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:01.445658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:01.445658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:01.445719 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:01.445939 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:01.445940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:01.445940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:01.446080 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:01.446244 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.446347 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.446421 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:09:01.448072 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:01.448225 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:01.449568 8176 tablet_service.cc:1467] Processing CreateTablet for tablet ff446c487c0941238ab505d23b7310d0 (DEFAULT_TABLE table=test-table [id=bfa1914881ef43379a8a455868adb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.449574 8036 tablet_service.cc:1467] Processing CreateTablet for tablet ff446c487c0941238ab505d23b7310d0 (DEFAULT_TABLE table=test-table [id=bfa1914881ef43379a8a455868adb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.449688 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff446c487c0941238ab505d23b7310d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.449690 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff446c487c0941238ab505d23b7310d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.451031 8411 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Bootstrap starting.
I20260924 03:09:01.451084 8412 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Bootstrap starting.
I20260924 03:09:01.451380 8412 tablet_bootstrap.cc:654] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.451411 8411 tablet_bootstrap.cc:654] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.451993 8411 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: No bootstrap required, opened a new log
I20260924 03:09:01.452039 8411 ts_tablet_manager.cc:1397] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:01.452073 8316 tablet_service.cc:1467] Processing CreateTablet for tablet ff446c487c0941238ab505d23b7310d0 (DEFAULT_TABLE table=test-table [id=bfa1914881ef43379a8a455868adb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.452179 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff446c487c0941238ab505d23b7310d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.452170 8411 raft_consensus.cc:348] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.452229 8411 raft_consensus.cc:374] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.452248 8411 raft_consensus.cc:729] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.452291 8411 consensus_queue.cc:260] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.452364 8411 ts_tablet_manager.cc:1428] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.453454 8410 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Bootstrap starting.
I20260924 03:09:01.453925 8410 tablet_bootstrap.cc:654] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.454370 8412 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: No bootstrap required, opened a new log
I20260924 03:09:01.454411 8412 ts_tablet_manager.cc:1397] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:09:01.454499 8412 raft_consensus.cc:348] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.454556 8412 raft_consensus.cc:374] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.454576 8412 raft_consensus.cc:729] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75815b3e01064b56810d64939382a32c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.454622 8412 consensus_queue.cc:260] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.454723 8412 ts_tablet_manager.cc:1428] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.455066 8410 tablet_bootstrap.cc:492] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: No bootstrap required, opened a new log
I20260924 03:09:01.455112 8410 ts_tablet_manager.cc:1397] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:01.455245 8410 raft_consensus.cc:348] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.455300 8410 raft_consensus.cc:374] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.455319 8410 raft_consensus.cc:729] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0164b2a35d34e6d96e78a8b77566a9c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.455358 8410 consensus_queue.cc:260] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.455435 8410 ts_tablet_manager.cc:1428] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.779969 8421 raft_consensus.cc:482] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:01.780089 8421 raft_consensus.cc:504] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.780263 8421 leader_election.cc:288] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.780499 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff446c487c0941238ab505d23b7310d0" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" is_pre_election: true
I20260924 03:09:01.780499 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff446c487c0941238ab505d23b7310d0" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" is_pre_election: true
I20260924 03:09:01.780609 8056 raft_consensus.cc:2454] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:01.780609 8336 raft_consensus.cc:2454] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 0.
I20260924 03:09:01.780762 8128 leader_election.cc:302] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:01.780885 8421 raft_consensus.cc:2781] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:01.780937 8421 raft_consensus.cc:482] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:01.780961 8421 raft_consensus.cc:3037] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.781652 8421 raft_consensus.cc:504] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.781787 8421 leader_election.cc:288] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Requested vote from peers e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:01.781955 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff446c487c0941238ab505d23b7310d0" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
I20260924 03:09:01.781948 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff446c487c0941238ab505d23b7310d0" candidate_uuid: "75815b3e01064b56810d64939382a32c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a61f20cbc4e5e9c8460a92bae271d"
I20260924 03:09:01.782037 8056 raft_consensus.cc:3037] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.782038 8336 raft_consensus.cc:3037] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:01.782662 8336 raft_consensus.cc:2454] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:01.782701 8056 raft_consensus.cc:2454] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75815b3e01064b56810d64939382a32c in term 1.
I20260924 03:09:01.782868 8128 leader_election.cc:302] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75815b3e01064b56810d64939382a32c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:01.782969 8421 raft_consensus.cc:2781] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:01.783020 8421 raft_consensus.cc:686] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 1 LEADER]: Becoming Leader. State: Replica: 75815b3e01064b56810d64939382a32c, State: Running, Role: LEADER
I20260924 03:09:01.783098 8421 consensus_queue.cc:237] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.783651 7894 catalog_manager.cc:5495] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c reported cstate change: term changed from 0 to 1, leader changed from <none> to 75815b3e01064b56810d64939382a32c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "75815b3e01064b56810d64939382a32c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:01.811316 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:01.809081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.0.98.2:46855)
0924 03:09:01.809139 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:01.809141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:01.809181 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:09:01.809248 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:01.809249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:01.809259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:01.809318 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:01.809427 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.809861 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.810649 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:01.810837 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:01.810910 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:01.811060 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:01.811096 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:01.811154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:01.811226 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:09:01.811243 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:09:01.811965 8336 raft_consensus.cc:1261] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.811966 8056 raft_consensus.cc:1261] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Refusing update from remote peer 75815b3e01064b56810d64939382a32c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:01.812212 8418 consensus_queue.cc:1035] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.812386 8418 consensus_queue.cc:1035] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:01.814558 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:01.814672 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:01.815272 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Sending DeleteTablet for 3 replicas of tablet ff446c487c0941238ab505d23b7310d0
I20260924 03:09:01.815478 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet ff446c487c0941238ab505d23b7310d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260924 03:09:01.815516 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet ff446c487c0941238ab505d23b7310d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45792
I20260924 03:09:01.815503 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet ff446c487c0941238ab505d23b7310d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37132
I20260924 03:09:01.815563 8434 tablet_replica.cc:330] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: stopping tablet replica
I20260924 03:09:01.815580 8435 tablet_replica.cc:330] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: stopping tablet replica
I20260924 03:09:01.815580 8436 tablet_replica.cc:330] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: stopping tablet replica
I20260924 03:09:01.815606 8434 raft_consensus.cc:2229] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:01.815614 8435 raft_consensus.cc:2229] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:01.815616 8436 raft_consensus.cc:2229] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:01.815642 8434 raft_consensus.cc:2258] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.815657 8435 raft_consensus.cc:2258] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.815670 8436 raft_consensus.cc:2258] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:01.815905 8434 ts_tablet_manager.cc:1905] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.815897 8435 ts_tablet_manager.cc:1905] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.815999 8436 ts_tablet_manager.cc:1905] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:01.817329 8435 ts_tablet_manager.cc:1918] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:01.817365 8436 ts_tablet_manager.cc:1918] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:01.817384 8435 log.cc:1196] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/ff446c487c0941238ab505d23b7310d0
I20260924 03:09:01.817417 8436 log.cc:1196] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/ff446c487c0941238ab505d23b7310d0
I20260924 03:09:01.817407 8434 ts_tablet_manager.cc:1918] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:01.817449 8434 log.cc:1196] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/ff446c487c0941238ab505d23b7310d0
I20260924 03:09:01.817619 8435 ts_tablet_manager.cc:1939] T ff446c487c0941238ab505d23b7310d0 P c0164b2a35d34e6d96e78a8b77566a9c: Deleting consensus metadata
I20260924 03:09:01.817643 8436 ts_tablet_manager.cc:1939] T ff446c487c0941238ab505d23b7310d0 P 75815b3e01064b56810d64939382a32c: Deleting consensus metadata
I20260924 03:09:01.817642 8434 ts_tablet_manager.cc:1939] T ff446c487c0941238ab505d23b7310d0 P e08a61f20cbc4e5e9c8460a92bae271d: Deleting consensus metadata
I20260924 03:09:01.817890 7885 catalog_manager.cc:4841] TS c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431): tablet ff446c487c0941238ab505d23b7310d0 (table test-table [id=bfa1914881ef43379a8a455868adb21d]) successfully deleted
I20260924 03:09:01.817922 7884 catalog_manager.cc:4841] TS e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057): tablet ff446c487c0941238ab505d23b7310d0 (table test-table [id=bfa1914881ef43379a8a455868adb21d]) successfully deleted
I20260924 03:09:01.817984 7884 catalog_manager.cc:4841] TS 75815b3e01064b56810d64939382a32c (127.0.98.2:46855): tablet ff446c487c0941238ab505d23b7310d0 (table test-table [id=bfa1914881ef43379a8a455868adb21d]) successfully deleted
I20260924 03:09:01.817941 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:01.818077 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:01.819228 8036 tablet_service.cc:1467] Processing CreateTablet for tablet c25d221fa3914170bae9217055aaa76b (DEFAULT_TABLE table=test-table [id=0f39661394d440ea880d8925091f887d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.819238 8176 tablet_service.cc:1467] Processing CreateTablet for tablet c25d221fa3914170bae9217055aaa76b (DEFAULT_TABLE table=test-table [id=0f39661394d440ea880d8925091f887d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.819257 8316 tablet_service.cc:1467] Processing CreateTablet for tablet c25d221fa3914170bae9217055aaa76b (DEFAULT_TABLE table=test-table [id=0f39661394d440ea880d8925091f887d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:01.819324 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25d221fa3914170bae9217055aaa76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.819324 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25d221fa3914170bae9217055aaa76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.819350 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25d221fa3914170bae9217055aaa76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:01.820225 8410 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Bootstrap starting.
I20260924 03:09:01.820428 8412 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Bootstrap starting.
I20260924 03:09:01.820551 8410 tablet_bootstrap.cc:654] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.820722 8412 tablet_bootstrap.cc:654] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.820932 8411 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: Bootstrap starting.
I20260924 03:09:01.821118 8410 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: No bootstrap required, opened a new log
I20260924 03:09:01.821167 8410 ts_tablet_manager.cc:1397] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:09:01.821238 8411 tablet_bootstrap.cc:654] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:01.821321 8410 raft_consensus.cc:348] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.821391 8410 raft_consensus.cc:374] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.821413 8410 raft_consensus.cc:729] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0164b2a35d34e6d96e78a8b77566a9c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.821460 8410 consensus_queue.cc:260] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.821550 8410 ts_tablet_manager.cc:1428] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.821832 8411 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: No bootstrap required, opened a new log
I20260924 03:09:01.821879 8411 ts_tablet_manager.cc:1397] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:01.821938 8412 tablet_bootstrap.cc:492] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: No bootstrap required, opened a new log
I20260924 03:09:01.821964 8411 raft_consensus.cc:348] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.821980 8412 ts_tablet_manager.cc:1397] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:09:01.822005 8411 raft_consensus.cc:374] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.822017 8411 raft_consensus.cc:729] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.822042 8411 consensus_queue.cc:260] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.822084 8412 raft_consensus.cc:348] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.822129 8411 ts_tablet_manager.cc:1428] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:01.822152 8412 raft_consensus.cc:374] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:01.822171 8412 raft_consensus.cc:729] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75815b3e01064b56810d64939382a32c, State: Initialized, Role: FOLLOWER
I20260924 03:09:01.822233 8412 consensus_queue.cc:260] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:01.822326 8412 ts_tablet_manager.cc:1428] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:02.204483 8466 raft_consensus.cc:482] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:02.204582 8466 raft_consensus.cc:504] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:02.204738 8466 leader_election.cc:288] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75815b3e01064b56810d64939382a32c (127.0.98.2:46855), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:02.204936 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25d221fa3914170bae9217055aaa76b" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75815b3e01064b56810d64939382a32c" is_pre_election: true
I20260924 03:09:02.204963 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25d221fa3914170bae9217055aaa76b" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" is_pre_election: true
I20260924 03:09:02.205030 8336 raft_consensus.cc:2454] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 0.
I20260924 03:09:02.205053 8196 raft_consensus.cc:2454] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 0.
I20260924 03:09:02.205178 7989 leader_election.cc:302] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0164b2a35d34e6d96e78a8b77566a9c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:02.205260 8466 raft_consensus.cc:2781] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:02.205304 8466 raft_consensus.cc:482] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:02.205322 8466 raft_consensus.cc:3037] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:02.206022 8466 raft_consensus.cc:504] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:02.206138 8466 leader_election.cc:288] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 election: Requested vote from peers 75815b3e01064b56810d64939382a32c (127.0.98.2:46855), c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431)
I20260924 03:09:02.206290 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25d221fa3914170bae9217055aaa76b" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75815b3e01064b56810d64939382a32c"
I20260924 03:09:02.206305 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25d221fa3914170bae9217055aaa76b" candidate_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0164b2a35d34e6d96e78a8b77566a9c"
I20260924 03:09:02.206367 8336 raft_consensus.cc:3037] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:02.206384 8196 raft_consensus.cc:3037] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:02.206910 8336 raft_consensus.cc:2454] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 1.
I20260924 03:09:02.206918 8196 raft_consensus.cc:2454] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a61f20cbc4e5e9c8460a92bae271d in term 1.
I20260924 03:09:02.207067 7989 leader_election.cc:302] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0164b2a35d34e6d96e78a8b77566a9c, e08a61f20cbc4e5e9c8460a92bae271d; no voters:
I20260924 03:09:02.207186 8466 raft_consensus.cc:2781] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:02.207294 8466 raft_consensus.cc:686] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 1 LEADER]: Becoming Leader. State: Replica: e08a61f20cbc4e5e9c8460a92bae271d, State: Running, Role: LEADER
I20260924 03:09:02.207347 8466 consensus_queue.cc:237] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260924 03:09:02.207929 7894 catalog_manager.cc:5495] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d reported cstate change: term changed from 0 to 1, leader changed from <none> to e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a61f20cbc4e5e9c8460a92bae271d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:02.262374 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:02.260285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.98.1:42057)
0924 03:09:02.260401 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:02.260402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:02.260410 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:02.260431 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:02.260432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:02.260449 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:02.260504 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:02.260671 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:02.261135 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:02.261749 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:02.261948 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:02.261995 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:02.262115 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:02.262145 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:02.262219 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:02.262275 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:09:02.262295 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260924 03:09:02.264285 8336 raft_consensus.cc:1261] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Refusing update from remote peer e08a61f20cbc4e5e9c8460a92bae271d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:02.264304 8196 raft_consensus.cc:1261] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Refusing update from remote peer e08a61f20cbc4e5e9c8460a92bae271d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:02.264489 8493 consensus_queue.cc:1035] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75815b3e01064b56810d64939382a32c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:02.264576 8466 consensus_queue.cc:1035] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0164b2a35d34e6d96e78a8b77566a9c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:02.273272 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:02.273345 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:02.273849 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6220237b55b4f279eebc5c26cee83fb: Sending DeleteTablet for 3 replicas of tablet c25d221fa3914170bae9217055aaa76b
I20260924 03:09:02.274048 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet c25d221fa3914170bae9217055aaa76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260924 03:09:02.274075 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet c25d221fa3914170bae9217055aaa76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45792
I20260924 03:09:02.274115 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet c25d221fa3914170bae9217055aaa76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37132
I20260924 03:09:02.274147 8436 tablet_replica.cc:330] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: stopping tablet replica
I20260924 03:09:02.274209 8436 raft_consensus.cc:2229] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:02.274219 8435 tablet_replica.cc:330] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: stopping tablet replica
I20260924 03:09:02.274228 8434 tablet_replica.cc:330] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d: stopping tablet replica
I20260924 03:09:02.274240 8436 raft_consensus.cc:2258] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:02.274266 8435 raft_consensus.cc:2229] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:02.274276 8434 raft_consensus.cc:2229] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:02.274297 8435 raft_consensus.cc:2258] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:02.274320 8434 raft_consensus.cc:2258] T c25d221fa3914170bae9217055aaa76b P e08a61f20cbc4e5e9c8460a92bae271d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:02.274416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7971
I20260924 03:09:02.274544 8436 ts_tablet_manager.cc:1905] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:02.275185 8435 ts_tablet_manager.cc:1905] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:02.276021 8436 ts_tablet_manager.cc:1918] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:02.276060 8436 log.cc:1196] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/c25d221fa3914170bae9217055aaa76b
I20260924 03:09:02.276222 8436 ts_tablet_manager.cc:1939] T c25d221fa3914170bae9217055aaa76b P 75815b3e01064b56810d64939382a32c: Deleting consensus metadata
I20260924 03:09:02.276472 7884 catalog_manager.cc:4841] TS 75815b3e01064b56810d64939382a32c (127.0.98.2:46855): tablet c25d221fa3914170bae9217055aaa76b (table test-table [id=0f39661394d440ea880d8925091f887d]) successfully deleted
I20260924 03:09:02.276818 8435 ts_tablet_manager.cc:1918] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:09:02.276865 8435 log.cc:1196] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/c25d221fa3914170bae9217055aaa76b
I20260924 03:09:02.277048 8435 ts_tablet_manager.cc:1939] T c25d221fa3914170bae9217055aaa76b P c0164b2a35d34e6d96e78a8b77566a9c: Deleting consensus metadata
I20260924 03:09:02.277467 7885 catalog_manager.cc:4841] TS c0164b2a35d34e6d96e78a8b77566a9c (127.0.98.3:40431): tablet c25d221fa3914170bae9217055aaa76b (table test-table [id=0f39661394d440ea880d8925091f887d]) successfully deleted
W20260924 03:09:02.282207 7884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42057 (error 108)
I20260924 03:09:02.282370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8111
I20260924 03:09:02.283056 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:02.282503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42057 (local address 127.0.0.1:38314)
0924 03:09:02.282945 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:02.282999 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":42,"threads_started":1}
W20260924 03:09:02.283275 7884 catalog_manager.cc:4584] TS e08a61f20cbc4e5e9c8460a92bae271d (127.0.98.1:42057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c25d221fa3914170bae9217055aaa76b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42057: connect: Connection refused (error 111)
I20260924 03:09:02.287936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8251
I20260924 03:09:02.292411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 7867
2026-09-24T03:09:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3561 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:02 dist-test-slave-jtnz krb5kdc[8505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:02 dist-test-slave-jtnz krb5kdc[8505](info): set up 2 sockets
Sep 24 03:09:02 dist-test-slave-jtnz krb5kdc[8505](info): commencing operation
krb5kdc: starting...
W20260924 03:09:03.258296 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:09:03 dist-test-slave-jtnz krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:09:03.364248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:03.435711 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:03.435877 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:03.435896 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:03.436959 8518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:03.436990 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:03.437001 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:03.437013 8518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:03.437024 8518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:03.438460 8518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:03.438623 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:03.438755 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:03.443002 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.442981 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:03.442992 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:03.443269 8518 server_base.cc:1034] running on GCE node
I20260924 03:09:03.443482 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:03.443737 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:03.444900 8518 hybrid_clock.cc:648] HybridClock initialized: now 1790219343444878 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T03:09:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1158 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:03 dist-test-slave-jtnz krb5kdc[8531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:03 dist-test-slave-jtnz krb5kdc[8531](info): set up 2 sockets
Sep 24 03:09:03 dist-test-slave-jtnz krb5kdc[8531](info): commencing operation
krb5kdc: starting...
W20260924 03:09:04.410487 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:09:04.509894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:04.582240 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:04.582492 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:04.582528 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:04.583621 8544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:04.583695 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:04.583719 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:04.583748 8544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:04.583776 8544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:04.585083 8544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:04.585312 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:04.585503 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:04.589653 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.589780 8544 server_base.cc:1034] running on GCE node
W20260924 03:09:04.589653 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.589641 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.590094 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:04.590423 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:04.591584 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790219344591581 us; error 36 us; skew 500 ppm
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:04.597115 8544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:09:04.597492 8544 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260924 03:09:04.597704 8544 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:04.597762 8544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:04.597893 8544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:04.598861 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "534900fe5d264de0a441bf25932967cd"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "f12eb83213eafbdd3c7070905eb51f79"
server_key_iv: "8d209346db2146ad37a2fb046b073b2c"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.599190 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "534900fe5d264de0a441bf25932967cd"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "f12eb83213eafbdd3c7070905eb51f79"
server_key_iv: "8d209346db2146ad37a2fb046b073b2c"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.600389 8544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:09:04.601044 8559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.601192 8544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:04.601286 8544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "534900fe5d264de0a441bf25932967cd"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "f12eb83213eafbdd3c7070905eb51f79"
server_key_iv: "8d209346db2146ad37a2fb046b073b2c"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.601362 8544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:04.611933 8544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:04.613603 8544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:04.613746 8544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:04.617339 8544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37199
I20260924 03:09:04.617401 8611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37199 every 8 connection(s)
I20260924 03:09:04.617815 8544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:04.618451 8612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:04.620452 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Bootstrap starting.
I20260924 03:09:04.620968 8612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:04.621186 8612 log.cc:824] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:04.621758 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: No bootstrap required, opened a new log
I20260924 03:09:04.623270 8612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:04.623386 8612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:04.623409 8612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534900fe5d264de0a441bf25932967cd, State: Initialized, Role: FOLLOWER
I20260924 03:09:04.623534 8612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:04.623610 8612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:04.623631 8612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:04.623687 8612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:04.623678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8544
I20260924 03:09:04.623776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:09:04.623940 392 external_mini_cluster.cc:1428] Setting key db04921839c0d1f7165a5aba749f3553
I20260924 03:09:04.624749 8612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:04.624850 8612 leader_election.cc:302] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534900fe5d264de0a441bf25932967cd; no voters:
I20260924 03:09:04.625007 8612 leader_election.cc:288] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:04.625068 8617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:04.625227 8617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 LEADER]: Becoming Leader. State: Replica: 534900fe5d264de0a441bf25932967cd, State: Running, Role: LEADER
I20260924 03:09:04.625252 8612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:04.625313 8617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:04.625730 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534900fe5d264de0a441bf25932967cd. Latest consensus state: current_term: 1 leader_uuid: "534900fe5d264de0a441bf25932967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260924 03:09:04.625811 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:04.625752 8618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534900fe5d264de0a441bf25932967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260924 03:09:04.625962 8618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: This master's current role is: LEADER
W20260924 03:09:04.627133 8632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:09:04.627224 8632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:09:04.627584 8626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:04.627766 8626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:04.629266 8626 catalog_manager.cc:1347] Generated new cluster ID: 27e19bbcb53948d0987ac530df39b6c7
I20260924 03:09:04.629319 8626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:04.635217 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.624390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.98.62:37199)
0924 03:09:04.624580 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.624589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.624621 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:09:04.624722 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.624729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.624780 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.624918 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.625930 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.627461 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.628475 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.629042 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.629159 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.631857 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.631860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.631861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.631871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.633648 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.634025 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.634028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.634028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.634122 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.634356 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.634357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.634357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.634599 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.634725 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.634854 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.635012 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260924 03:09:04.644446 8626 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:04.644843 8626 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:04.646462 8626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Generated new TSK 0
I20260924 03:09:04.646585 8626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:09:04.670055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:04.745719 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:04.745888 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:04.745908 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:04.746918 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:04.746954 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:04.746977 8638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:04.748205 8638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:04.748376 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:04.748517 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:04.752898 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.752938 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.752915 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.753208 8638 server_base.cc:1034] running on GCE node
I20260924 03:09:04.753394 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:04.753619 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:04.754757 8638 hybrid_clock.cc:648] HybridClock initialized: now 1790219344754744 us; error 26 us; skew 500 ppm
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:04.757069 8638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:09:04.757405 8638 webserver.cc:460] Webserver started at http://127.0.98.1:42047/ using document root <none> and password file <none>
I20260924 03:09:04.757567 8638 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:04.757604 8638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:04.757684 8638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:04.758599 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "ac4d57baf13d4a7d9d244294338d1f4f"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "b2be55babb0a4b7b5662e547c37b09cc"
server_key_iv: "ba7b00100686c199d2d85841bc2a95b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.758860 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ac4d57baf13d4a7d9d244294338d1f4f"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "b2be55babb0a4b7b5662e547c37b09cc"
server_key_iv: "ba7b00100686c199d2d85841bc2a95b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.759966 8638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:04.760589 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.760754 8638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:04.760847 8638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "ac4d57baf13d4a7d9d244294338d1f4f"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "b2be55babb0a4b7b5662e547c37b09cc"
server_key_iv: "ba7b00100686c199d2d85841bc2a95b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.760924 8638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:04.770753 8638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:04.772909 8638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:04.773061 8638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:04.773273 8638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:04.773545 8638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:04.773597 8638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.773634 8638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:04.773661 8638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.778810 8638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45377
I20260924 03:09:04.778863 8766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45377 every 8 connection(s)
I20260924 03:09:04.779124 8638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:04.783993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8638
I20260924 03:09:04.784083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:04.784299 392 external_mini_cluster.cc:1428] Setting key 98947f90912061517c48cf6de95123e6
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:04.788009 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.779715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:59213)
0924 03:09:04.779859 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.779916 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.780747 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.780986 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.780990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.781217 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.781366 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.781369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.782024 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.782025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.782790 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.782792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.783133 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.783134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.783247 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.783665 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.783672 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.785222 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.786754 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.786760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.786762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.787043 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.787221 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.787222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.787222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.787340 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.787612 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.787614 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.787719 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.787787 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.787882 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260924 03:09:04.788040 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.779734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59213 (local address 127.0.98.62:37199)
0924 03:09:04.779790 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.779792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.780002 (+ 210us) server_negotiation.cc:407] Connection header received
0924 03:09:04.780852 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.780853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.780864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.780921 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.781463 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.781939 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.782879 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.783062 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.783129 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.785308 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.785310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.785311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.785318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.786677 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.787116 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.787117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.787117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.787162 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.787420 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.787420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.787421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.787552 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.787622 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.787829 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.787906 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:09:04.788462 8767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260924 03:09:04.788554 8767 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:04.788721 8767 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260924 03:09:04.789170 8576 ts_manager.cc:194] Registered new tserver with Master: ac4d57baf13d4a7d9d244294338d1f4f (127.0.98.1:45377)
I20260924 03:09:04.789719 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59213
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:09:04.806493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:04.879042 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:04.879235 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:04.879267 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:04.880330 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:04.880394 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:04.880445 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:04.881891 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:04.882133 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:04.882346 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:04.886848 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.886862 8772 server_base.cc:1034] running on GCE node
W20260924 03:09:04.886848 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:04.886862 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:04.887275 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:04.887503 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:04.888655 8772 hybrid_clock.cc:648] HybridClock initialized: now 1790219344888642 us; error 25 us; skew 500 ppm
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:04.890934 8772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:09:04.891362 8772 webserver.cc:460] Webserver started at http://127.0.98.2:33021/ using document root <none> and password file <none>
I20260924 03:09:04.891587 8772 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:04.891649 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:04.891788 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:04.892711 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "e202c609daf94e1ea533887fad3056ac"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "8df031eec7098de596ae97ddcdb60e4d"
server_key_iv: "c6747fb9c43726909bfe94772e66cf8b"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.893051 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e202c609daf94e1ea533887fad3056ac"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "8df031eec7098de596ae97ddcdb60e4d"
server_key_iv: "c6747fb9c43726909bfe94772e66cf8b"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.894225 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:09:04.894882 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.895038 8772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:04.895123 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "e202c609daf94e1ea533887fad3056ac"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "8df031eec7098de596ae97ddcdb60e4d"
server_key_iv: "c6747fb9c43726909bfe94772e66cf8b"
server_key_version: "encryptionkey@0"
I20260924 03:09:04.895201 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:04.913122 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:04.914776 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:04.914913 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:04.915103 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:04.915350 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:04.915396 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.915426 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:04.915452 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:04.920573 8772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38937
I20260924 03:09:04.920614 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38937 every 8 connection(s)
I20260924 03:09:04.920868 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:09:04 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:04.928822 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.921518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37159 (local address 127.0.98.62:37199)
0924 03:09:04.921587 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:04.921590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:04.921745 (+ 155us) server_negotiation.cc:407] Connection header received
0924 03:09:04.922624 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:04.922624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:04.922633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:04.922678 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:04.923179 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.923620 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.924420 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.924578 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.924623 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.926895 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:04.926896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:04.926896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:04.926902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:04.927885 (+ 983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.928197 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.928197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.928198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.928239 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:04.928377 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:04.928378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:04.928378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:04.928457 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:04.928498 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:04.928711 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:09:04.928768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:09:04.928947 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:04.921483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:37159)
0924 03:09:04.921645 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:04.921674 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:04.922543 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:04.922731 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:04.922733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:04.922945 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:04.923122 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.923125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.923686 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.923686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.924357 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:04.924358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:04.924641 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:04.924641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:04.924722 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:04.925158 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:04.925165 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:04.926823 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:04.927955 (+ 1132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.927960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.927962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.928140 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.928279 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:04.928279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:04.928279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:04.928347 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:04.928498 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:04.928500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:04.928583 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 03:09:04.928672 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 03:09:04.928775 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260924 03:09:04.929354 8901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260924 03:09:04.929443 8901 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:04.929607 8901 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260924 03:09:04.929937 8576 ts_manager.cc:194] Registered new tserver with Master: e202c609daf94e1ea533887fad3056ac (127.0.98.2:38937)
I20260924 03:09:04.930279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8772
I20260924 03:09:04.930349 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37159
I20260924 03:09:04.930370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:04.930574 392 external_mini_cluster.cc:1428] Setting key a7da1bc4ed23a7cfbc84bdf7e79c2467
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:09:04.953050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:05.025214 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:05.025393 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:05.025432 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:05.026712 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:05.026768 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:05.026808 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:05.028098 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:05.028316 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:05.028523 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:05.032932 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.032922 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.032922 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.033005 8906 server_base.cc:1034] running on GCE node
I20260924 03:09:05.033313 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:05.033546 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:05.034683 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790219345034680 us; error 33 us; skew 500 ppm
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:05.036929 8906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:09:05.037353 8906 webserver.cc:460] Webserver started at http://127.0.98.3:44701/ using document root <none> and password file <none>
I20260924 03:09:05.037529 8906 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:05.037563 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:05.037638 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:05.038560 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "f43ea4bbb7f24df181300e774bb7e36c"
format_stamp: "Formatted at 2026-09-24 03:09:05 on dist-test-slave-jtnz"
server_key: "4a60ef62084757ad1b1ecbc112f62938"
server_key_iv: "fd8ee5e0a1658eaf62baaa8315c4c160"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.038817 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f43ea4bbb7f24df181300e774bb7e36c"
format_stamp: "Formatted at 2026-09-24 03:09:05 on dist-test-slave-jtnz"
server_key: "4a60ef62084757ad1b1ecbc112f62938"
server_key_iv: "fd8ee5e0a1658eaf62baaa8315c4c160"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.039912 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:05.040534 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.040728 8906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:05.040803 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "f43ea4bbb7f24df181300e774bb7e36c"
format_stamp: "Formatted at 2026-09-24 03:09:05 on dist-test-slave-jtnz"
server_key: "4a60ef62084757ad1b1ecbc112f62938"
server_key_iv: "fd8ee5e0a1658eaf62baaa8315c4c160"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.040851 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:05.067387 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:05.069559 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:05.069655 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:05.069799 8906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:05.070072 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:05.070103 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.070122 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:05.070132 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.075158 8906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46741
I20260924 03:09:05.075223 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46741 every 8 connection(s)
I20260924 03:09:05.075485 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:05.076746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 8906
I20260924 03:09:05.076826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:05.077004 392 external_mini_cluster.cc:1428] Setting key 604ac548226d7d873134e1eb38dc0312
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:05.083477 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.076077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49973 (local address 127.0.98.62:37199)
0924 03:09:05.076143 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.076145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.076278 (+ 133us) server_negotiation.cc:407] Connection header received
0924 03:09:05.077104 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.077105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.077117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.077173 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.077818 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.078297 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.079132 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.079319 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.079362 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.081093 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:05.081094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:05.081095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:05.081101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:05.082503 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:05.082814 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:05.082815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:05.082815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:05.082859 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:05.083056 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:05.083057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:05.083057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:05.083140 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:05.083184 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.083371 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.083422 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:09:05.083680 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.076036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:49973)
0924 03:09:05.076180 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.076208 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.076996 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.077234 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.077238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.077493 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.077737 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.077742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.078359 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.078360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.079062 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.079064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.079378 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.079379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.079457 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.079805 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.079811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.081019 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:05.082563 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:05.082567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:05.082569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:05.082754 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:05.082912 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:05.082913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:05.082913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:05.082994 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:05.083188 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:05.083191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:05.083289 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 03:09:05.083359 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:09:05.083452 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260924 03:09:05.084059 9035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260924 03:09:05.084136 9035 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:05.084239 9035 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260924 03:09:05.084527 8576 ts_manager.cc:194] Registered new tserver with Master: f43ea4bbb7f24df181300e774bb7e36c (127.0.98.3:46741)
I20260924 03:09:05.084805 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49973
I20260924 03:09:05.088485 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:09:05.146759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8638
I20260924 03:09:05.151041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8772
I20260924 03:09:05.154644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8906
I20260924 03:09:05.158365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 8544
I20260924 03:09:05.162915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--webserver_interface=127.0.98.62
--webserver_port=35907
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:05.233757 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:05.233906 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:05.233932 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:05.234841 9044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:05.234870 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:05.234884 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:05.234894 9044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:05.234903 9044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:05.236061 9044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:05.236255 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:05.236429 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:05.240541 9049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.240545 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.240696 9044 server_base.cc:1034] running on GCE node
W20260924 03:09:05.240541 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.240967 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:05.241276 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:05.242430 9044 hybrid_clock.cc:648] HybridClock initialized: now 1790219345242415 us; error 25 us; skew 500 ppm
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:05.244755 9044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:09:05.245159 9044 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260924 03:09:05.245321 9044 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:05.245368 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:05.246608 9044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:09:05.247151 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.247311 9044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:05.247370 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "534900fe5d264de0a441bf25932967cd"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "f12eb83213eafbdd3c7070905eb51f79"
server_key_iv: "8d209346db2146ad37a2fb046b073b2c"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.247601 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:05.275120 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:05.277024 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:05.277148 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:05.280639 9044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37199
I20260924 03:09:05.280686 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37199 every 8 connection(s)
I20260924 03:09:05.281165 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:05.282325 9112 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:09:05.282903 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Bootstrap starting.
I20260924 03:09:05.286787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9044
I20260924 03:09:05.287034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=42047
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:09:05.290825 9112 log.cc:824] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:05.292760 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:09:05.293066 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Bootstrap complete.
I20260924 03:09:05.296758 9112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:05.297116 9112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534900fe5d264de0a441bf25932967cd, State: Initialized, Role: FOLLOWER
I20260924 03:09:05.297349 9112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:05.297437 9112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:05.297484 9112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:05.297538 9112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:09:05.298792 9112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:05.298897 9112 leader_election.cc:302] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534900fe5d264de0a441bf25932967cd; no voters:
I20260924 03:09:05.299046 9112 leader_election.cc:288] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:09:05.299118 9116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:09:05.299254 9116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [term 2 LEADER]: Becoming Leader. State: Replica: 534900fe5d264de0a441bf25932967cd, State: Running, Role: LEADER
I20260924 03:09:05.299261 9112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:05.299906 9116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260924 03:09:05.300408 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "534900fe5d264de0a441bf25932967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260924 03:09:05.300485 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:05.300638 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534900fe5d264de0a441bf25932967cd. Latest consensus state: current_term: 2 leader_uuid: "534900fe5d264de0a441bf25932967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534900fe5d264de0a441bf25932967cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260924 03:09:05.300657 9131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:05.300913 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:05.301326 9131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:05.301501 9131 catalog_manager.cc:1259] Loaded cluster ID: 27e19bbcb53948d0987ac530df39b6c7
I20260924 03:09:05.301558 9131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:05.302453 9131 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:05.302634 9131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 534900fe5d264de0a441bf25932967cd: Loaded TSK: 0
I20260924 03:09:05.302845 9131 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:09:05.383590 9114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:05.383775 9114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:05.383805 9114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:05.384778 9114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:05.384845 9114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:05.384882 9114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:05.386106 9114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42047
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:05.386339 9114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:05.386557 9114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:05.390879 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.390909 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.390980 9114 server_base.cc:1034] running on GCE node
W20260924 03:09:05.391129 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.391383 9114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:05.391652 9114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:05.392814 9114 hybrid_clock.cc:648] HybridClock initialized: now 1790219345392804 us; error 27 us; skew 500 ppm
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:05.395314 9114 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:09:05.395784 9114 webserver.cc:460] Webserver started at http://127.0.98.1:42047/ using document root <none> and password file <none>
I20260924 03:09:05.395995 9114 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:05.396070 9114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:05.397258 9114 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:05.397801 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.397949 9114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:05.398017 9114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "ac4d57baf13d4a7d9d244294338d1f4f"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "b2be55babb0a4b7b5662e547c37b09cc"
server_key_iv: "ba7b00100686c199d2d85841bc2a95b7"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.398334 9114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:05.410735 9114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:05.412719 9114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:05.412846 9114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:05.413072 9114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:05.413391 9114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:05.413434 9114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.413452 9114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:05.413471 9114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.420257 9114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45377
I20260924 03:09:05.420300 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45377 every 8 connection(s)
I20260924 03:09:05.420640 9114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:05.421993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9114
I20260924 03:09:05.422312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38937
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=33021
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.430393 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.421239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:57789)
0924 03:09:05.421440 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.421482 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.422043 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.422651 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.422655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.423337 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.423570 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.423576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.424855 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.424856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.425942 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.425944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.426923 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.426925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.427083 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.427589 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.427599 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.430201 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
W20260924 03:09:05.430565 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.421355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57789 (local address 127.0.98.62:37199)
0924 03:09:05.421561 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.421567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.421591 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:09:05.422303 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.422310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.422361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.422515 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.423690 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.424763 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.426026 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.426690 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.426832 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.430354 (+ 3522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57789: BlockingRecv error: recv got EOF from 127.0.98.1:57789 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20260924 03:09:05.430878 9262 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.435993 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.431158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:43063)
0924 03:09:05.431270 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.431280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.431324 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.431458 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.431459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.431614 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.431667 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.431668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.432539 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.432540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.433244 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.433247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.433658 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.433658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.433714 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.434104 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.434108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.435928 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
W20260924 03:09:05.436065 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.431211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43063 (local address 127.0.98.62:37199)
0924 03:09:05.431247 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.431249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.431334 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:09:05.431363 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.431364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.431377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.431446 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.431735 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.432450 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.433327 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.433592 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.433650 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.436007 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43063: BlockingRecv error: recv got EOF from 127.0.98.1:43063 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.440897 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.436460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:55943)
0924 03:09:05.436491 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.436499 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.436525 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.436648 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.436648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.436766 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.436790 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.436791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.437562 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.437563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.438289 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.438291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.438734 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.438735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.438781 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.439216 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.439221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.440837 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 03:09:05.440991 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.436511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55943 (local address 127.0.98.62:37199)
0924 03:09:05.436542 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.436543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.436566 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:09:05.436583 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.436583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.436591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.436635 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.436850 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.437500 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.438363 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.438653 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.438724 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.440922 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55943: BlockingRecv error: recv got EOF from 127.0.98.1:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:09:05.441162 9262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:09:05.501119 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:05.501284 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:05.501307 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:05.502560 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:05.502599 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:05.502624 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:05.503911 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33021
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:05.504074 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:05.504225 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:05.508637 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.508671 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.508656 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.508774 9266 server_base.cc:1034] running on GCE node
I20260924 03:09:05.508998 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:05.509173 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:05.510299 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790219345510287 us; error 28 us; skew 500 ppm
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:05.512596 9266 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:09:05.512981 9266 webserver.cc:460] Webserver started at http://127.0.98.2:33021/ using document root <none> and password file <none>
I20260924 03:09:05.513195 9266 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:05.513247 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:05.514534 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:05.515031 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.515213 9266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:05.515303 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "e202c609daf94e1ea533887fad3056ac"
format_stamp: "Formatted at 2026-09-24 03:09:04 on dist-test-slave-jtnz"
server_key: "8df031eec7098de596ae97ddcdb60e4d"
server_key_iv: "c6747fb9c43726909bfe94772e66cf8b"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.515570 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:05.524197 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:05.525671 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:05.525810 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:05.525995 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:05.526291 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:05.526335 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.526371 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:05.526392 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.531708 9266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38937
I20260924 03:09:05.531771 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38937 every 8 connection(s)
I20260924 03:09:05.532053 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.537329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9266
I20260924 03:09:05.537562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46741
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=44701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:05.537808 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.532657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33839 (local address 127.0.98.62:37199)
0924 03:09:05.532700 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.532701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.532889 (+ 188us) server_negotiation.cc:407] Connection header received
0924 03:09:05.533266 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.533267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.533278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.533325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.534069 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.534539 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.535352 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.535509 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.535552 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.537761 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33839: BlockingRecv error: recv got EOF from 127.0.98.2:33839 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:09:05.537850 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.532648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:33839)
0924 03:09:05.532790 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.532817 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.533193 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.533385 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.533388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.533846 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.534006 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.534009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.534604 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.534604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.535284 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.535286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.535569 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.535569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.535652 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.536025 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.536031 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.537700 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260924 03:09:05.538022 9396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.542162 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.538240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:48137)
0924 03:09:05.538277 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.538286 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.538318 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.538439 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.538440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.538551 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.538587 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.538588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.539120 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.539120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.539556 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.539557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.539915 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.539916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.539984 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.540360 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.540364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.542106 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 03:09:05.542323 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.538263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48137 (local address 127.0.98.62:37199)
0924 03:09:05.538299 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.538301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.538308 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:05.538369 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.538369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.538376 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.538437 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.538658 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.539043 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.539659 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.539830 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.539916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.542246 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48137: BlockingRecv error: recv got EOF from 127.0.98.2:48137 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.546502 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.542376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:37597)
0924 03:09:05.542494 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.542503 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.542533 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.542674 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.542675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.542763 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.542783 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.542784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.543574 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.543574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.543984 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.543985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.544420 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.544421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.544450 (+ 29us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.544816 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.544820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.546464 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260924 03:09:05.546630 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.542478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37597 (local address 127.0.98.62:37199)
0924 03:09:05.542523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.542524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.542532 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:05.542595 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.542596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.542605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.542667 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.542844 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.543499 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.544053 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.544343 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.544410 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.546557 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37597: BlockingRecv error: recv got EOF from 127.0.98.2:37597 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 03:09:05.546684 9396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:09:05.612594 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:05.612757 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:05.612777 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:05.613788 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:05.613823 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:05.613847 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:05.615156 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44701
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:05.615371 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:05.615566 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:05.620016 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.620082 9399 server_base.cc:1034] running on GCE node
W20260924 03:09:05.620029 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:05.620028 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:05.620435 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:05.620664 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:05.621794 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790219345621786 us; error 26 us; skew 500 ppm
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219345, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:05.624143 9399 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:09:05.624541 9399 webserver.cc:460] Webserver started at http://127.0.98.3:44701/ using document root <none> and password file <none>
I20260924 03:09:05.624747 9399 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:05.624812 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:05.625939 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:09:05.626477 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.626650 9399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.626740 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "f43ea4bbb7f24df181300e774bb7e36c"
format_stamp: "Formatted at 2026-09-24 03:09:05 on dist-test-slave-jtnz"
server_key: "4a60ef62084757ad1b1ecbc112f62938"
server_key_iv: "fd8ee5e0a1658eaf62baaa8315c4c160"
server_key_version: "encryptionkey@0"
I20260924 03:09:05.627033 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:05.640363 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:05.642568 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:05.642711 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:05.642920 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:05.643198 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:05.643244 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.643280 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:05.643304 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:05.648608 9399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46741
I20260924 03:09:05.648669 9528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46741 every 8 connection(s)
I20260924 03:09:05.648960 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:05.651937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9399
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:09:05.656479 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.649583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34047 (local address 127.0.98.62:37199)
0924 03:09:05.649624 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.649626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.649814 (+ 188us) server_negotiation.cc:407] Connection header received
0924 03:09:05.650341 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.650342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.650356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.650426 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.651268 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.651941 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.653096 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.653402 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.653466 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.656418 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34047: BlockingRecv error: recv got EOF from 127.0.98.3:34047 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:09:05.656639 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.649557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:34047)
0924 03:09:05.649701 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.649736 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.650234 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.650492 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.650495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.651019 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.651188 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.651192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.652057 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.652058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.652993 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.652996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.653487 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.653488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.653647 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.654086 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.654095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.656365 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260924 03:09:05.656848 9529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 03:09:05.659873 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.652429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.0.98.62:37199)
0924 03:09:05.652587 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.652589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.652597 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:05.652618 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.652619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.652629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.652707 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.653390 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.654110 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.655084 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.655379 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.655447 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.656692 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:05.656698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:05.656698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:05.656729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:05.658564 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:05.658879 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:05.658882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:05.658882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:05.658995 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:05.659207 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:05.659208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:05.659208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:05.659414 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:05.659559 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:05.659668 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:09:05.659772 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.661255 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.657048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:45243)
0924 03:09:05.657162 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.657170 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.657207 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.657335 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.657336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.657465 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.657514 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.657516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.658170 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.658171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.658890 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.658892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.659203 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.659204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.659265 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.659640 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.659645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.661217 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 03:09:05.661378 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.657110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45243 (local address 127.0.98.62:37199)
0924 03:09:05.657149 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.657150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.657198 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:09:05.657272 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.657272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.657280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.657330 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.657605 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.658095 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.658959 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.659118 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.659201 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.661309 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45243: BlockingRecv error: recv got EOF from 127.0.98.3:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:05 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:05.665067 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:05.661473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:57769)
0924 03:09:05.661498 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:05.661508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:05.661541 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:05.661674 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:05.661675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:05.661764 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:05.661785 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.661786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.662356 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.662356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.662799 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:05.662800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.663303 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.663303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:05.663336 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.663622 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:05.663626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:05.665031 (+ 1405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 03:09:05.665177 9529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:09:05.665187 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:05.661543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57769 (local address 127.0.98.62:37199)
0924 03:09:05.661591 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:05.661593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:05.661603 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:05.661622 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:05.661622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:05.661628 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:05.661671 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:05.661832 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.662299 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.662863 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:05.663228 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:05.663296 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:05.665121 (+ 1825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57769: BlockingRecv error: recv got EOF from 127.0.98.3:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:06.445926 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:06.441534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:55451)
0924 03:09:06.441670 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:06.441679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:06.441728 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:06.441863 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:06.441864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:06.442048 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:06.442143 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.442149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.442926 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.442927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.443444 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.443446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.443808 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.443809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.443877 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.444261 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:06.444265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:06.445869 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260924 03:09:06.446043 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:06.441610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55451 (local address 127.0.98.62:37199)
0924 03:09:06.441709 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:06.441719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:06.441728 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:06.441771 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:06.441771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:06.441780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:06.441843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:06.442265 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.442845 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.443531 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.443713 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.443809 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.445963 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55451: BlockingRecv error: recv got EOF from 127.0.98.1:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:06.551256 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:06.546987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:39711)
0924 03:09:06.547096 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:06.547106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:06.547161 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:06.547275 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:06.547276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:06.547411 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:06.547456 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.547457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.548094 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.548095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.548588 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.548590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.548995 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.548997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.549060 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.549526 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:06.549530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:06.551196 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
W20260924 03:09:06.551376 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:06.547057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39711 (local address 127.0.98.62:37199)
0924 03:09:06.547098 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:06.547100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:06.547150 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:09:06.547201 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:06.547202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:06.547212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:06.547261 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:06.547513 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.548017 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.548647 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.548908 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.548993 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.551282 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39711: BlockingRecv error: recv got EOF from 127.0.98.2:39711 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:06 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:06.669943 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:06.665583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:41653)
0924 03:09:06.665732 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:06.665742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:06.665796 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:06.665912 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:06.665913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:06.666043 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:06.666086 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.666087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.666661 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.666662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.667305 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:06.667307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.667577 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.667578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:06.667609 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.668070 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:06.668074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:06.669854 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
W20260924 03:09:06.669988 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:06.665682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41653 (local address 127.0.98.62:37199)
0924 03:09:06.665741 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:06.665743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:06.665796 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:09:06.665836 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:06.665836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:06.665845 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:06.665894 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:06.666140 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.666605 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.667362 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:06.667523 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:06.667570 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:06.669920 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41653: BlockingRecv error: recv got EOF from 127.0.98.3:41653 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:07.450830 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.446481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:50293)
0924 03:09:07.446628 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:07.446644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:07.446691 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:07.446843 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:07.446844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:07.446996 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:07.447043 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.447044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.447637 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.447638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.448169 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.448170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.448471 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.448472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.448507 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.448917 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:07.448921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:07.450729 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260924 03:09:07.450845 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:07.446575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50293 (local address 127.0.98.62:37199)
0924 03:09:07.446679 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.446681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.446690 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:07.446768 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.446769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.446778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.446838 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:07.447098 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.447576 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.448235 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.448408 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.448454 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.450784 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50293: BlockingRecv error: recv got EOF from 127.0.98.1:50293 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:07.556192 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.551666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:48297)
0924 03:09:07.551828 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:07.551839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:07.551907 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:07.552026 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:07.552027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:07.552194 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:07.552236 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.552239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.552875 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.552876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.553382 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.553383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.553743 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.553745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.553801 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.554334 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:07.554339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:07.556104 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
W20260924 03:09:07.556241 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:07.551753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48297 (local address 127.0.98.62:37199)
0924 03:09:07.551824 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.551826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.551889 (+ 63us) server_negotiation.cc:407] Connection header received
0924 03:09:07.551968 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.551969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.551979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.552022 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:07.552316 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.552795 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.553467 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.553660 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.553742 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.556179 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48297: BlockingRecv error: recv got EOF from 127.0.98.2:48297 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:07 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:07.674716 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:07.670558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:33329)
0924 03:09:07.670708 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:07.670741 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:07.670799 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:07.670935 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:07.670936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:07.671092 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:07.671136 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.671138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.671785 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.671786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.672298 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:07.672299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.672661 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.672662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:07.672724 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.673126 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:07.673130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:07.674635 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
W20260924 03:09:07.674767 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:07.670703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33329 (local address 127.0.98.62:37199)
0924 03:09:07.670771 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:07.670773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:07.670782 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:07.670866 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:07.670867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:07.670880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:07.670922 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:07.671211 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.671702 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.672381 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:07.672576 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:07.672659 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:07.674708 (+ 2049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33329: BlockingRecv error: recv got EOF from 127.0.98.3:33329 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:08.455361 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:08.451487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:60271)
0924 03:09:08.451621 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:08.451630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:08.451676 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:08.451809 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:08.451810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:08.451971 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:08.452016 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.452018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.452592 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.452592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.453065 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.453067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.453335 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.453336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.453368 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.453806 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:08.453810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:08.455312 (+ 1502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260924 03:09:08.455487 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:08.451541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60271 (local address 127.0.98.62:37199)
0924 03:09:08.451663 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:08.451667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:08.451713 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:09:08.451732 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:08.451733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:08.451743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:08.451791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:08.452075 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.452536 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.453125 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.453282 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.453325 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.455385 (+ 2060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60271: BlockingRecv error: recv got EOF from 127.0.98.1:60271 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:08.561050 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:08.556706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:50407)
0924 03:09:08.556867 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:08.556878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:08.556975 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:08.557132 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:08.557133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:08.557295 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:08.557339 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.557340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.557984 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.557986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.558551 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.558553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.558947 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.558948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.559006 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.559405 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:08.559409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:08.560955 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260924 03:09:08.561084 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:08.556795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50407 (local address 127.0.98.62:37199)
0924 03:09:08.556865 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:08.556868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:08.556953 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:09:08.557044 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:08.557045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:08.557057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:08.557128 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:08.557413 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.557901 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.558631 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.558845 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.558931 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.561022 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50407: BlockingRecv error: recv got EOF from 127.0.98.2:50407 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:08 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:08.679533 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:08.675279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:59947)
0924 03:09:08.675432 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:08.675454 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:08.675521 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:08.675645 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:08.675646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:08.675799 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:08.675840 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.675842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.676470 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.676471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.676963 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:08.676964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.677293 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.677294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:08.677372 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.677865 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:08.677869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:08.679431 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
W20260924 03:09:08.679566 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:08.675330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59947 (local address 127.0.98.62:37199)
0924 03:09:08.675392 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:08.675394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:08.675507 (+ 113us) server_negotiation.cc:407] Connection header received
0924 03:09:08.675570 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:08.675571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:08.675584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:08.675636 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:08.675913 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.676367 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.677042 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:08.677216 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:08.677292 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:08.679505 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59947: BlockingRecv error: recv got EOF from 127.0.98.3:59947 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:09.460100 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.455970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:50087)
0924 03:09:09.456101 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:09.456117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:09.456162 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:09.456284 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:09.456285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:09.456429 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:09.456472 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.456473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.457029 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.457030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.457542 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.457543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.457842 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.457843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.457877 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.458340 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:09.458344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:09.459989 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
W20260924 03:09:09.460189 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:09.456037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50087 (local address 127.0.98.62:37199)
0924 03:09:09.456152 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.456154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.456169 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:09:09.456204 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.456205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.456214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.456269 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.456529 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.456970 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.457605 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.457786 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.457830 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.460107 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50087: BlockingRecv error: recv got EOF from 127.0.98.1:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:09.565773 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.561504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:58229)
0924 03:09:09.561640 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:09.561656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:09.561708 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:09.561830 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:09.561830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:09.561981 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:09.562026 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.562028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.562711 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.562712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.563228 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.563230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.563552 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.563553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.563586 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.564105 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:09.564121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:09.565721 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260924 03:09:09.565877 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:09.561575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58229 (local address 127.0.98.62:37199)
0924 03:09:09.561637 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.561638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.561717 (+ 79us) server_negotiation.cc:407] Connection header received
0924 03:09:09.561743 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.561744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.561753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.561812 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.562104 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.562646 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.563321 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.563497 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.563542 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.565807 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58229: BlockingRecv error: recv got EOF from 127.0.98.2:58229 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:09 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:09.687175 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:09.679979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:38609)
0924 03:09:09.680148 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:09.680158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:09.680207 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:09.680306 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:09.680307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:09.680445 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:09.680488 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.680490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.681066 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.681067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.681557 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:09.681558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.681841 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.681842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:09.681878 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.685460 (+ 3582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:09.685464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:09.687126 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
W20260924 03:09:09.687295 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:09.680135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38609 (local address 127.0.98.62:37199)
0924 03:09:09.680214 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:09.680216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:09.680224 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:09.680244 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:09.680244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:09.680254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:09.680302 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:09.680541 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.681006 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.681617 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:09.681779 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:09.681836 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:09.687219 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38609: BlockingRecv error: recv got EOF from 127.0.98.3:38609 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:10.465108 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:10.460599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:54783)
0924 03:09:10.460727 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:10.460736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:10.460797 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:10.460953 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:10.460953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:10.461106 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:10.461154 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.461156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.461767 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.461767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.462301 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.462302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.462649 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.462650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.462691 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.463192 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:10.463197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:10.465017 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260924 03:09:10.465173 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:10.460675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54783 (local address 127.0.98.62:37199)
0924 03:09:10.460815 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:10.460817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:10.460824 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:10.460843 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:10.460843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:10.460852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:10.460938 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:10.461217 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.461706 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.462367 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.462586 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.462638 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.465093 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54783: BlockingRecv error: recv got EOF from 127.0.98.1:54783 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:10.569973 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:10.566227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:49919)
0924 03:09:10.566362 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:10.566372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:10.566430 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:10.566537 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:10.566538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:10.566670 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:10.566714 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.566716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.567233 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.567234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.567668 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.567669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.567939 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.567939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.567973 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.568403 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:10.568407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:10.569921 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
W20260924 03:09:10.570070 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:10.566332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49919 (local address 127.0.98.62:37199)
0924 03:09:10.566385 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:10.566387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:10.566416 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:09:10.566480 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:10.566480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:10.566490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:10.566532 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:10.566762 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.567177 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.567731 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.567884 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.567926 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.570004 (+ 2078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49919: BlockingRecv error: recv got EOF from 127.0.98.2:49919 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:10 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:10.691982 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:10.687631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:47897)
0924 03:09:10.687773 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:10.687783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:10.687835 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:10.687944 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:10.687945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:10.688084 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:10.688128 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.688130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.688770 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.688771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.689281 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:10.689283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.689603 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.689604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:10.689640 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.690136 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:10.690141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:10.691877 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260924 03:09:10.692028 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:10.687794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47897 (local address 127.0.98.62:37199)
0924 03:09:10.687839 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:10.687840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:10.687850 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:10.687870 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:10.687871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:10.687881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:10.687926 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:10.688187 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.688705 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.689355 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:10.689538 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:10.689592 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:10.691957 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47897: BlockingRecv error: recv got EOF from 127.0.98.3:47897 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:11.469810 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:11.465632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:56875)
0924 03:09:11.465779 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:11.465789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:11.465839 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:11.465968 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:11.465968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:11.466112 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:11.466156 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.466157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.466763 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.466764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.467266 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.467268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.467603 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.467604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.467642 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.468048 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:11.468064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:11.469747 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
W20260924 03:09:11.469934 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:11.465685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56875 (local address 127.0.98.62:37199)
0924 03:09:11.465799 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:11.465801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:11.465827 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:09:11.465889 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:11.465890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:11.465901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:11.465946 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:11.466228 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.466704 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.467349 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.467545 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.467590 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.469831 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56875: BlockingRecv error: recv got EOF from 127.0.98.1:56875 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:11.574935 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:11.570527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:56133)
0924 03:09:11.570656 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:11.570666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:11.570731 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:11.570856 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:11.570857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:11.571049 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:11.571095 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.571097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.571798 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.571800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.572333 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.572334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.572727 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.572728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.572785 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.573199 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:11.573203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:11.574872 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
W20260924 03:09:11.574994 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:11.570618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56133 (local address 127.0.98.62:37199)
0924 03:09:11.570657 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:11.570659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:11.570717 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:09:11.570789 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:11.570790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:11.570801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:11.570849 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:11.571172 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.571704 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.572445 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.572643 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.572725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.574954 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56133: BlockingRecv error: recv got EOF from 127.0.98.2:56133 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:11 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:11.696858 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:11.692544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:32793)
0924 03:09:11.692738 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:11.692750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:11.692840 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:11.692975 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:11.692976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:11.693154 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:11.693197 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.693199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.693842 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.693843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.694360 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:11.694361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.694697 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.694697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:11.694750 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.695187 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:11.695192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:11.696798 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":112,"threads_started":1}
W20260924 03:09:11.696978 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:11.692657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32793 (local address 127.0.98.62:37199)
0924 03:09:11.692732 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:11.692734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:11.692821 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:09:11.692892 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:11.692892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:11.692906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:11.692961 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:11.693274 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.693761 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.694444 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:11.694620 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:11.694698 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:11.696882 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32793: BlockingRecv error: recv got EOF from 127.0.98.3:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:12.474725 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.470454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:44853)
0924 03:09:12.470573 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:12.470585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:12.470644 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:12.470809 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:12.470810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:12.470985 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:12.471031 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.471032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.471647 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.471648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.472155 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.472157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.472458 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.472459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.472496 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.472933 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:12.472937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:12.474637 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260924 03:09:12.474778 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:12.470513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44853 (local address 127.0.98.62:37199)
0924 03:09:12.470600 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:12.470602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:12.470631 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:09:12.470708 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:12.470709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:12.470722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:12.470791 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:12.471097 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.471580 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.472221 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.472400 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.472447 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.474709 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44853: BlockingRecv error: recv got EOF from 127.0.98.1:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":42,"threads_started":1}
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:12.579509 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.575391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:54867)
0924 03:09:12.575539 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:12.575550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:12.575630 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:12.575787 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:12.575788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:12.575941 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:12.575986 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.575987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.576588 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.576589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.577071 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.577073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.577384 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.577391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.577446 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.577904 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:12.577908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:12.579422 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260924 03:09:12.579556 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:12.575474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54867 (local address 127.0.98.62:37199)
0924 03:09:12.575542 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:12.575544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:12.575614 (+ 70us) server_negotiation.cc:407] Connection header received
0924 03:09:12.575709 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:12.575710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:12.575722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:12.575779 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:12.576059 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.576504 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.577143 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.577311 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.577383 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.579497 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54867: BlockingRecv error: recv got EOF from 127.0.98.2:54867 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:12 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:12.701578 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:12.697323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:50787)
0924 03:09:12.697478 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:12.697488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:12.697539 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:12.697722 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:12.697723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:12.697887 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:12.697934 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.697935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.698578 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.698580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.699036 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:12.699038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.699393 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.699394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:12.699469 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.699832 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:12.699836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:12.701524 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
W20260924 03:09:12.701673 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:12.697557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50787 (local address 127.0.98.62:37199)
0924 03:09:12.697622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:12.697623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:12.697629 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:09:12.697646 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:12.697646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:12.697656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:12.697713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:12.698007 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.698508 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.699109 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:12.699315 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:12.699384 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:12.701608 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50787: BlockingRecv error: recv got EOF from 127.0.98.3:50787 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:13.479626 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:13.475137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:44051)
0924 03:09:13.475274 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:13.475284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:13.475328 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:13.475476 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:13.475477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:13.475627 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:13.475671 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.475672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.476267 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.476268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.476814 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.476816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.477125 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.477126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.477162 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.477579 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:13.477583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:13.479534 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260924 03:09:13.479684 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:13.475236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44051 (local address 127.0.98.62:37199)
0924 03:09:13.475361 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:13.475362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:13.475369 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:13.475388 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:13.475389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:13.475398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:13.475460 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:13.475731 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.476209 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.476882 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.477067 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.477114 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.479612 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44051: BlockingRecv error: recv got EOF from 127.0.98.1:44051 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:13.584224 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:13.579922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:40007)
0924 03:09:13.580035 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:13.580051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:13.580098 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:13.580203 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:13.580204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:13.580344 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:13.580386 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.580387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.581117 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.581117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.581635 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.581636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.581947 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.581948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.581985 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.582449 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:13.582454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:13.584173 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260924 03:09:13.584349 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:13.579990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40007 (local address 127.0.98.62:37199)
0924 03:09:13.580047 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:13.580049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:13.580094 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:09:13.580151 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:13.580152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:13.580161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:13.580193 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:13.580437 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.581053 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.581702 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.581885 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.581940 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.584252 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40007: BlockingRecv error: recv got EOF from 127.0.98.2:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:13 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:13.705905 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:13.702041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:32921)
0924 03:09:13.702216 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:13.702227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:13.702290 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:13.702446 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:13.702447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:13.702597 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:13.702644 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.702646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.703248 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.703249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.703736 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:13.703737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.704015 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.704016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:13.704052 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.704413 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:13.704417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:13.705848 (+ 1431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260924 03:09:13.706017 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:13.702101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32921 (local address 127.0.98.62:37199)
0924 03:09:13.702202 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:13.702204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:13.702301 (+ 97us) server_negotiation.cc:407] Connection header received
0924 03:09:13.702367 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:13.702368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:13.702379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:13.702432 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:13.702705 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.703188 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.703796 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:13.703962 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:13.704004 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:13.705930 (+ 1926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32921: BlockingRecv error: recv got EOF from 127.0.98.3:32921 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:14.484346 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:14.480158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:54001)
0924 03:09:14.480277 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:14.480289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:14.480350 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:14.480488 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:14.480490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:14.480642 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:14.480687 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.480688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.481284 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.481285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.481793 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.481794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.482103 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.482104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.482139 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.482591 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:14.482595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:14.484293 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
W20260924 03:09:14.484483 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:14.480220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54001 (local address 127.0.98.62:37199)
0924 03:09:14.480372 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:14.480374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:14.480384 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:14.480405 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:14.480406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:14.480416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:14.480471 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:14.480749 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.481225 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.481868 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.482046 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.482092 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.484397 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54001: BlockingRecv error: recv got EOF from 127.0.98.1:54001 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:14.588583 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:14.584692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:48917)
0924 03:09:14.584801 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:14.584811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:14.584857 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:14.584998 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:14.584999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:14.585159 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:14.585204 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.585205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.585771 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.585772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.586315 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.586318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.586600 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.586601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.586634 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.587036 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:14.587041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:14.588514 (+ 1473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
W20260924 03:09:14.588704 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:14.584753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48917 (local address 127.0.98.62:37199)
0924 03:09:14.584838 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:14.584840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:14.584850 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:14.584924 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:14.584924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:14.584934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:14.584980 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:14.585261 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.585714 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.586381 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.586541 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.586585 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.588625 (+ 2040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48917: BlockingRecv error: recv got EOF from 127.0.98.2:48917 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:14 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:14.713758 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:14.706404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:57125)
0924 03:09:14.706583 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:14.706594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:14.706676 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:14.706817 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:14.706818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:14.706985 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:14.707045 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.707047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.707658 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.707659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.708158 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:14.708160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.708463 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.708464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:14.708521 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.712163 (+ 3642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:14.712168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:14.713678 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
W20260924 03:09:14.713819 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:14.706488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57125 (local address 127.0.98.62:37199)
0924 03:09:14.706565 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:14.706567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:14.706660 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:09:14.706729 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:14.706730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:14.706743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:14.706809 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:14.707125 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.707587 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.708235 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:14.708401 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:14.708465 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:14.713744 (+ 5279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57125: BlockingRecv error: recv got EOF from 127.0.98.3:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:15.489073 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:15.484788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:35475)
0924 03:09:15.484946 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:15.484957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:15.485005 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:15.485166 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:15.485166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:15.485321 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:15.485365 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.485367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.485971 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.485971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.486572 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.486574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.486861 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.486861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.486903 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.487315 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:15.487320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:15.489018 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
W20260924 03:09:15.489203 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:15.484902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35475 (local address 127.0.98.62:37199)
0924 03:09:15.485041 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:15.485043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:15.485052 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:15.485082 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:15.485082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:15.485092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:15.485153 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:15.485435 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.485904 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.486630 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.486804 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.486852 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.489101 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35475: BlockingRecv error: recv got EOF from 127.0.98.1:35475 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:15.593436 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:15.589245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:47605)
0924 03:09:15.589388 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:15.589403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:15.589486 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:15.589637 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:15.589638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:15.589788 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:15.589834 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.589835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.590449 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.590450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.590944 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.590946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.591244 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.591245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.591282 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.591801 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:15.591805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:15.593375 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260924 03:09:15.593531 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:15.589314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47605 (local address 127.0.98.62:37199)
0924 03:09:15.589383 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:15.589385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:15.589469 (+ 84us) server_negotiation.cc:407] Connection header received
0924 03:09:15.589556 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:15.589557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:15.589569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:15.589613 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:15.589891 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.590380 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.591022 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.591187 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.591233 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.593464 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47605: BlockingRecv error: recv got EOF from 127.0.98.2:47605 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:15 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:15.718703 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:15.714332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:58055)
0924 03:09:15.714467 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:15.714480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:15.714535 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:15.714676 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:15.714677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:15.714839 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:15.714882 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.714884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.715496 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.715497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.716016 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:15.716017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.716345 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.716345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:15.716397 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.716926 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:15.716931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:15.718620 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
W20260924 03:09:15.718748 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:15.714451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58055 (local address 127.0.98.62:37199)
0924 03:09:15.714508 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:15.714510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:15.714517 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:09:15.714591 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:15.714592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:15.714607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:15.714670 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:15.714954 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.715423 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.716094 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:15.716273 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:15.716345 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:15.718687 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58055: BlockingRecv error: recv got EOF from 127.0.98.3:58055 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:16.493810 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:16.489537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:35923)
0924 03:09:16.489673 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:16.489684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:16.489739 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:16.489961 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:16.489963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:16.490113 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:16.490160 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.490161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.490791 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.490792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.491328 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.491329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.491639 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.491640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.491678 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.492106 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:16.492110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:16.493709 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260924 03:09:16.493848 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:16.489671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35923 (local address 127.0.98.62:37199)
0924 03:09:16.489826 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:16.489827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:16.489836 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:09:16.489865 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:16.489865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:16.489875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:16.489944 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:16.490254 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.490728 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.491396 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.491576 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.491625 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.493783 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35923: BlockingRecv error: recv got EOF from 127.0.98.1:35923 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:16.598416 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:16.593890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:38845)
0924 03:09:16.594090 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:16.594103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:16.594171 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:16.594328 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:16.594328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:16.594507 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:16.594550 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.594552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.595226 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.595227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.595720 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.595722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.596123 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.596124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.596180 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.596671 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:16.596675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:16.598316 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260924 03:09:16.598455 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:16.593988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38845 (local address 127.0.98.62:37199)
0924 03:09:16.594072 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:16.594074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:16.594151 (+ 77us) server_negotiation.cc:407] Connection header received
0924 03:09:16.594248 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:16.594248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:16.594260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:16.594314 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:16.594643 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.595136 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.595848 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.596038 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.596124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.598399 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38845: BlockingRecv error: recv got EOF from 127.0.98.2:38845 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:16 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:16.723433 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:16.719113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:60825)
0924 03:09:16.719258 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:16.719268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:16.719349 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:16.719502 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:16.719503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:16.719656 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:16.719700 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.719702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.720306 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.720307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.720822 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:16.720824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.721160 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.721160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:16.721218 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.721652 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:16.721656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:16.723343 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260924 03:09:16.723480 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:16.719178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60825 (local address 127.0.98.62:37199)
0924 03:09:16.719251 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:16.719254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:16.719337 (+ 83us) server_negotiation.cc:407] Connection header received
0924 03:09:16.719425 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:16.719426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:16.719440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:16.719496 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:16.719775 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.720225 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.720876 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:16.721083 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:16.721147 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:16.723418 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60825: BlockingRecv error: recv got EOF from 127.0.98.3:60825 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:17.498898 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.494374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:46403)
0924 03:09:17.494653 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:17.494663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:17.494718 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:17.494821 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:17.494821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:17.494981 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:17.495046 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.495047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.495951 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.495952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.496487 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.496488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.496768 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.496769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.496803 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.497276 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:17.497281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:17.498832 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
W20260924 03:09:17.499053 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:17.494425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46403 (local address 127.0.98.62:37199)
0924 03:09:17.494661 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:17.494663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:17.494702 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:09:17.494757 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:17.494757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:17.494766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:17.494818 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:17.495101 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.495888 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.496549 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.496714 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.496756 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.498928 (+ 2172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46403: BlockingRecv error: recv got EOF from 127.0.98.1:46403 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":184,"threads_started":1}
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:17.603729 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.599003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:51515)
0924 03:09:17.599107 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:17.599123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:17.599175 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:17.599309 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:17.599309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:17.599521 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:17.599571 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.599573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.600258 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.600259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.600861 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.600863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.601188 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.601188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.601228 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.601643 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:17.601647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:17.603671 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
W20260924 03:09:17.603790 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:17.599061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51515 (local address 127.0.98.62:37199)
0924 03:09:17.599118 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:17.599121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:17.599189 (+ 68us) server_negotiation.cc:407] Connection header received
0924 03:09:17.599246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:17.599247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:17.599256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:17.599303 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:17.599630 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.600195 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.600923 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.601087 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.601135 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.603746 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51515: BlockingRecv error: recv got EOF from 127.0.98.2:51515 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:17 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:17.728255 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:17.724021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:38841)
0924 03:09:17.724185 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:17.724210 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:17.724284 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:17.724448 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:17.724449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:17.724598 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:17.724640 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.724641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.725248 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.725250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.725730 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:17.725731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.726085 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.726085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:17.726143 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.726659 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:17.726663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:17.728172 (+ 1509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
W20260924 03:09:17.728293 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:17.724105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38841 (local address 127.0.98.62:37199)
0924 03:09:17.724171 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:17.724173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:17.724264 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:09:17.724348 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:17.724349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:17.724361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:17.724429 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:17.724709 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.725171 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.725816 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:17.726008 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:17.726086 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:17.728236 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38841: BlockingRecv error: recv got EOF from 127.0.98.3:38841 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:18.503981 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.499352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:58107)
0924 03:09:18.499528 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:18.499542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:18.499594 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:18.499772 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:18.499772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:18.499963 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:18.500012 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.500013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.500687 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.500688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.501307 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.501309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.501634 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.501634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.501693 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.502162 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:18.502166 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:18.503891 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":127,"threads_started":1}
W20260924 03:09:18.504022 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:18.499392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58107 (local address 127.0.98.62:37199)
0924 03:09:18.499540 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.499542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.499580 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:09:18.499641 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.499641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.499652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.499738 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.500074 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.500612 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.501373 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.501544 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.501593 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.503959 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58107: BlockingRecv error: recv got EOF from 127.0.98.1:58107 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:18.608349 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.604252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:34689)
0924 03:09:18.604389 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:18.604403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:18.604455 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:18.604567 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:18.604568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:18.604708 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:18.604752 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.604754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.605303 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.605304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.605770 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.605771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.606099 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.606100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.606139 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.606556 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:18.606561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:18.608286 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
W20260924 03:09:18.608421 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:18.604352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34689 (local address 127.0.98.62:37199)
0924 03:09:18.604399 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.604401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.604457 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:09:18.604504 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.604505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.604514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.604551 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.604807 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.605246 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.605838 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.606039 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.606087 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.608380 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34689: BlockingRecv error: recv got EOF from 127.0.98.2:34689 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:18 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:18.733281 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:18.728878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:56371)
0924 03:09:18.729017 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:18.729031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:18.729105 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:18.729257 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:18.729258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:18.729416 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:18.729461 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.729463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.730102 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.730104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.730658 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:18.730660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.731013 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.731014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:18.731073 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.731571 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:18.731575 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:18.733207 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260924 03:09:18.733389 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:18.728932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56371 (local address 127.0.98.62:37199)
0924 03:09:18.728991 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:18.728993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:18.729086 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:09:18.729172 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:18.729173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:18.729185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:18.729249 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:18.729534 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.730015 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.730749 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:18.730931 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:18.731012 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:18.733297 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56371: BlockingRecv error: recv got EOF from 127.0.98.3:56371 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:19.508421 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:19.504379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:38351)
0924 03:09:19.504510 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:19.504519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:19.504564 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:19.504684 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:19.504684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:19.504825 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:19.504870 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.504871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.505437 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.505437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.505905 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.505907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.506283 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.506284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.506325 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.506824 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:19.506829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:19.508370 (+ 1541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260924 03:09:19.508549 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:19.504443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38351 (local address 127.0.98.62:37199)
0924 03:09:19.504543 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:19.504545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:19.504553 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:19.504617 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:19.504618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:19.504627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:19.504680 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:19.504925 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.505381 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.505967 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.506152 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.506261 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.508450 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38351: BlockingRecv error: recv got EOF from 127.0.98.1:38351 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:19.613422 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:19.608928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:46385)
0924 03:09:19.609077 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:19.609089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:19.609153 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:19.609317 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:19.609318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:19.609492 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:19.609554 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.609556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.610375 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.610376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.610873 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.610875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.611195 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.611196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.611232 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.611647 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:19.611652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:19.613330 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
W20260924 03:09:19.613457 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:19.609110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46385 (local address 127.0.98.62:37199)
0924 03:09:19.609173 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:19.609175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:19.609183 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:19.609204 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:19.609205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:19.609215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:19.609266 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:19.609641 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.610301 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.610944 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.611134 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.611190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.613401 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46385: BlockingRecv error: recv got EOF from 127.0.98.2:46385 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:19 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:19.741076 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:19.733673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:60435)
0924 03:09:19.733823 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:19.733832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:19.733895 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:19.734010 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:19.734010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:19.734148 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:19.734209 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.734211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.734840 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.734840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.735331 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:19.735332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.735611 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.735612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:19.735646 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.739461 (+ 3815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:19.739466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:19.741025 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
W20260924 03:09:19.741170 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:19.733708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60435 (local address 127.0.98.62:37199)
0924 03:09:19.733769 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:19.733771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:19.733893 (+ 122us) server_negotiation.cc:407] Connection header received
0924 03:09:19.733947 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:19.733948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:19.733959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:19.733997 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:19.734265 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.734780 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.735392 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:19.735556 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:19.735602 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:19.741104 (+ 5502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60435: BlockingRecv error: recv got EOF from 127.0.98.3:60435 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:09:20 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:20 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:20.513301 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.508913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:44023)
0924 03:09:20.509048 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:20.509058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:20.509121 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:20.509309 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:20.509310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:20.509499 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:20.509557 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.509559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.510158 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.510160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.510673 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.510675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.511015 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.511016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.511076 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.511581 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:20.511586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:20.513248 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":43,"threads_started":1}
W20260924 03:09:20.513430 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:20.508987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44023 (local address 127.0.98.62:37199)
0924 03:09:20.509142 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:20.509145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:20.509155 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:09:20.509215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:20.509216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:20.509227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:20.509284 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:20.509623 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.510081 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.510754 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.510936 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.511013 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.513322 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44023: BlockingRecv error: recv got EOF from 127.0.98.1:44023 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
Sep 24 03:09:20 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:09:20 dist-test-slave-jtnz krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:09:20.618388 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.614148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:48793)
0924 03:09:20.614291 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:20.614301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:20.614351 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:20.614469 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:20.614470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:20.614612 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:20.614655 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.614656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.615260 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.615261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.615801 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.615805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.616212 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.616213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.616271 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.616664 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:20.616668 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:20.618328 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37199: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
W20260924 03:09:20.618530 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:20.614224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48793 (local address 127.0.98.62:37199)
0924 03:09:20.614282 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:20.614284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:20.614349 (+ 65us) server_negotiation.cc:407] Connection header received
0924 03:09:20.614395 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:20.614395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:20.614404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:20.614453 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:20.614706 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.615200 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.615877 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.616130 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.616194 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.618423 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48793: BlockingRecv error: recv got EOF from 127.0.98.2:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:09:20.658612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9114
I20260924 03:09:20.662911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9266
I20260924 03:09:20.666787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9399
I20260924 03:09:20.670387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9044
2026-09-24T03:09:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17218 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T03:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:20Z Disabled control of system clock
I20260924 03:09:20.696308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42893
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:09:20.770509 9596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:20.770692 9596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:20.770725 9596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:20.771782 9596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:20.771843 9596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:20.771867 9596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:20.771898 9596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:20.771922 9596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:20.773236 9596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:20.773469 9596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:20.773710 9596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:20.778059 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.778132 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.778047 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:20.778163 9596 server_base.cc:1034] running on GCE node
I20260924 03:09:20.778424 9596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:20.778680 9596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:20.779841 9596 hybrid_clock.cc:648] HybridClock initialized: now 1790219360779826 us; error 48 us; skew 500 ppm
I20260924 03:09:20.780887 9596 webserver.cc:460] Webserver started at http://127.0.98.62:43211/ using document root <none> and password file <none>
I20260924 03:09:20.781116 9596 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:20.781179 9596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:20.781303 9596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:20.782341 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "f62cb9585f554823bf7bfad22c43131a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bba12be7fcf75ab49796392a8b8f30ee"
server_key_iv: "a7c663d79a7b925d4b31a47187fa0554"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.782680 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "f62cb9585f554823bf7bfad22c43131a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bba12be7fcf75ab49796392a8b8f30ee"
server_key_iv: "a7c663d79a7b925d4b31a47187fa0554"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.783825 9596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:20.784440 9610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:20.784590 9596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:20.784672 9596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "f62cb9585f554823bf7bfad22c43131a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bba12be7fcf75ab49796392a8b8f30ee"
server_key_iv: "a7c663d79a7b925d4b31a47187fa0554"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.784747 9596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:20.807298 9596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:20.807548 9596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:20.807682 9596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:20.811537 9596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42893
I20260924 03:09:20.811568 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42893 every 8 connection(s)
I20260924 03:09:20.811862 9596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:20.812428 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:20.814725 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a: Bootstrap starting.
I20260924 03:09:20.815217 9663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:20.815425 9663 log.cc:824] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:20.815975 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a: No bootstrap required, opened a new log
I20260924 03:09:20.817282 9663 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 03:09:20.817391 9663 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:20.817410 9663 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62cb9585f554823bf7bfad22c43131a, State: Initialized, Role: FOLLOWER
I20260924 03:09:20.817493 9663 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 03:09:20.817562 9663 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:20.817590 9663 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:20.817639 9663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:20.818118 9663 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 03:09:20.818249 9663 leader_election.cc:302] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62cb9585f554823bf7bfad22c43131a; no voters:
I20260924 03:09:20.818415 9663 leader_election.cc:288] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:20.818460 9666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:20.818614 9666 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [term 1 LEADER]: Becoming Leader. State: Replica: f62cb9585f554823bf7bfad22c43131a, State: Running, Role: LEADER
I20260924 03:09:20.818660 9663 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:20.818704 9666 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } }
I20260924 03:09:20.819069 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62cb9585f554823bf7bfad22c43131a. Latest consensus state: current_term: 1 leader_uuid: "f62cb9585f554823bf7bfad22c43131a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } } }
I20260924 03:09:20.819216 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62cb9585f554823bf7bfad22c43131a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62cb9585f554823bf7bfad22c43131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42893 } } }
I20260924 03:09:20.819320 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:20.819227 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:20.819693 9675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:20.820149 9675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:20.820242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9596
I20260924 03:09:20.820318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:09:20.820518 392 external_mini_cluster.cc:1428] Setting key 918b01cdd6dd709ebdbc1300a1a51ac4
I20260924 03:09:20.822090 9675 catalog_manager.cc:1347] Generated new cluster ID: 5cdc3ea967204c6da206ec594f8c62de
I20260924 03:09:20.822141 9675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:20.826079 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.820915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.98.62:42893)
0924 03:09:20.821110 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:20.821117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:20.821161 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:09:20.821262 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:20.821268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:20.821358 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:20.821508 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:20.821707 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.823017 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.824107 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.824663 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.824760 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.825397 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:20.825401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:20.825401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:20.825574 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:20.825609 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:20.825620 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:20.825677 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:20.825764 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 03:09:20.825905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 03:09:20.837126 9675 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:20.837742 9675 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:20.850569 9675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62cb9585f554823bf7bfad22c43131a: Generated new TSK 0
I20260924 03:09:20.850728 9675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:20.861302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--builtin_ntp_servers=127.0.98.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:09:20.937237 9687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:20.937412 9687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:20.937433 9687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:20.938632 9687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:20.938666 9687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:20.938688 9687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:20.939929 9687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:20.940137 9687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:20.940389 9687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:20.944944 9693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.944950 9692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:20.944967 9695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:20.945128 9687 server_base.cc:1034] running on GCE node
I20260924 03:09:20.945399 9687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:20.945606 9687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:20.946736 9687 hybrid_clock.cc:648] HybridClock initialized: now 1790219360946725 us; error 28 us; skew 500 ppm
I20260924 03:09:20.947767 9687 webserver.cc:460] Webserver started at http://127.0.98.1:38061/ using document root <none> and password file <none>
I20260924 03:09:20.947976 9687 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:20.948015 9687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:20.948144 9687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:20.949090 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "b1705478e1cd4601ac4ab4c1e4931c2a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bda82b71bd1e1758db8f9d78503c7e96"
server_key_iv: "e21913bc8b0a18894bea189c1eb957f8"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.949399 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b1705478e1cd4601ac4ab4c1e4931c2a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bda82b71bd1e1758db8f9d78503c7e96"
server_key_iv: "e21913bc8b0a18894bea189c1eb957f8"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.950675 9687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:09:20.951391 9701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:20.951565 9687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:20.951629 9687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "b1705478e1cd4601ac4ab4c1e4931c2a"
format_stamp: "Formatted at 2026-09-24 03:09:20 on dist-test-slave-jtnz"
server_key: "bda82b71bd1e1758db8f9d78503c7e96"
server_key_iv: "e21913bc8b0a18894bea189c1eb957f8"
server_key_version: "encryptionkey@0"
I20260924 03:09:20.951711 9687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:20.962985 9687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:20.963246 9687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:20.963373 9687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:20.963567 9687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:20.963842 9687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:20.963901 9687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:20.963940 9687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:20.963966 9687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:20.969326 9687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45419
I20260924 03:09:20.969388 9814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45419 every 8 connection(s)
I20260924 03:09:20.969674 9687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:20.974025 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.970358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58227 (local address 127.0.98.62:42893)
0924 03:09:20.970412 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:20.970414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:20.970592 (+ 178us) server_negotiation.cc:407] Connection header received
0924 03:09:20.971012 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:20.971013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:20.971024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:20.971072 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:20.971619 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.972091 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.972974 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.973145 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.973192 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.973818 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:20.973819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:20.973819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:20.973823 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:20.973836 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:20.973839 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:20.973902 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:20.973949 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 03:09:20.973972 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:09:20.974270 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:20.970345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42893 (local address 127.0.98.1:58227)
0924 03:09:20.970495 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:20.970522 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:20.970935 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:20.971116 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:20.971120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:20.971127 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:20.971555 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.971558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.972164 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.972164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.972891 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:20.972893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:20.973221 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:20.973221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:20.973314 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:20.973706 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:20.973707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:20.973729 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:20.973747 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:20.973905 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:20.973907 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:20.973909 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:09:20.973951 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:09:20.974021 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260924 03:09:20.974776 9815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42893
I20260924 03:09:20.974865 9815 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:20.975015 9815 heartbeater.cc:511] Master 127.0.98.62:42893 requested a full tablet report, sending...
I20260924 03:09:20.975437 9627 ts_manager.cc:194] Registered new tserver with Master: b1705478e1cd4601ac4ab4c1e4931c2a (127.0.98.1:45419)
I20260924 03:09:20.975518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9687
I20260924 03:09:20.975600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:20.975759 392 external_mini_cluster.cc:1428] Setting key 9782015b97343d72f1a5b7527a1654bc
I20260924 03:09:20.976235 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58227
I20260924 03:09:20.976686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--builtin_ntp_servers=127.0.98.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:09:21.051508 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.051692 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.051723 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.052815 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.052884 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.052927 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:21.054250 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.054466 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.054716 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.059168 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.059202 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.059219 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.059270 9818 server_base.cc:1034] running on GCE node
I20260924 03:09:21.059573 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.059803 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.060966 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790219361060941 us; error 37 us; skew 500 ppm
I20260924 03:09:21.062003 9818 webserver.cc:460] Webserver started at http://127.0.98.2:35285/ using document root <none> and password file <none>
I20260924 03:09:21.062292 9818 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.062356 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.062486 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.063393 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "720331b5a1034f28bd08b7aa1edb446f"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e3860ee20d8d46f03e7ff3ec8fef545c"
server_key_iv: "780fd384da217f15aa81c7be9319988b"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.063721 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "720331b5a1034f28bd08b7aa1edb446f"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e3860ee20d8d46f03e7ff3ec8fef545c"
server_key_iv: "780fd384da217f15aa81c7be9319988b"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.064917 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:21.065562 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.065732 9818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:21.065816 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "720331b5a1034f28bd08b7aa1edb446f"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e3860ee20d8d46f03e7ff3ec8fef545c"
server_key_iv: "780fd384da217f15aa81c7be9319988b"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.065904 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.079998 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.080253 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.080400 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.080602 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.080899 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.080945 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.080976 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.080999 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.086414 9818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43321
I20260924 03:09:21.086453 9945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43321 every 8 connection(s)
I20260924 03:09:21.086696 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:21.090837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9818
I20260924 03:09:21.090917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:21.091115 392 external_mini_cluster.cc:1428] Setting key c9ac24c827a76cda1455d9c6a5c57e76
I20260924 03:09:21.091360 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.087356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46221 (local address 127.0.98.62:42893)
0924 03:09:21.087403 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.087405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.087611 (+ 206us) server_negotiation.cc:407] Connection header received
0924 03:09:21.088007 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.088008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.088017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.088066 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.088640 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.089109 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.089985 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.090158 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.090285 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.091039 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.091040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.091040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.091044 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.091055 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.091058 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.091094 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.091249 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.091279 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:09:21.091563 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.087344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42893 (local address 127.0.98.2:46221)
0924 03:09:21.087513 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.087539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.087932 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.088108 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.088111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.088118 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.088580 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.088584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.089177 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.089177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.089907 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.089909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.090284 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.090285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.090383 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.090833 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:21.090834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.090858 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:21.090877 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.091124 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.091130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.091134 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.091246 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.091365 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 03:09:21.091941 9946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42893
I20260924 03:09:21.092064 9946 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.092088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--builtin_ntp_servers=127.0.98.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:09:21.092250 9946 heartbeater.cc:511] Master 127.0.98.62:42893 requested a full tablet report, sending...
I20260924 03:09:21.092585 9627 ts_manager.cc:194] Registered new tserver with Master: 720331b5a1034f28bd08b7aa1edb446f (127.0.98.2:43321)
I20260924 03:09:21.093072 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46221
W20260924 03:09:21.165376 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.165534 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.165551 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.166582 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.166616 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.166637 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:21.167723 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.167881 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.168129 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.172261 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.172246 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.172242 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.172530 9949 server_base.cc:1034] running on GCE node
I20260924 03:09:21.172760 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.172989 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.174139 9949 hybrid_clock.cc:648] HybridClock initialized: now 1790219361174143 us; error 36 us; skew 500 ppm
I20260924 03:09:21.175319 9949 webserver.cc:460] Webserver started at http://127.0.98.3:34139/ using document root <none> and password file <none>
I20260924 03:09:21.175552 9949 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.175618 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.175751 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.176668 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "aac6b683ecc2405da3d12874277174ef"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "4ef5e4e6dd2b5dae72d00fd6f393419c"
server_key_iv: "e1147f516879bcdb22197569d529ab8d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.176994 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aac6b683ecc2405da3d12874277174ef"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "4ef5e4e6dd2b5dae72d00fd6f393419c"
server_key_iv: "e1147f516879bcdb22197569d529ab8d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.178153 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:09:21.178864 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.179019 9949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:21.179122 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "aac6b683ecc2405da3d12874277174ef"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "4ef5e4e6dd2b5dae72d00fd6f393419c"
server_key_iv: "e1147f516879bcdb22197569d529ab8d"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.179203 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.208108 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.208357 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.208492 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.208681 9949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.208957 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.209004 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.209041 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.209067 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.214309 9949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45817
I20260924 03:09:21.214349 10076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45817 every 8 connection(s)
I20260924 03:09:21.214630 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:21.216720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 9949
I20260924 03:09:21.216807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:21.216974 392 external_mini_cluster.cc:1428] Setting key 64dfceccf701778458fa25fcd9b96bb6
I20260924 03:09:21.219709 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.215287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59465 (local address 127.0.98.62:42893)
0924 03:09:21.215367 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.215369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.215528 (+ 159us) server_negotiation.cc:407] Connection header received
0924 03:09:21.215934 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.215935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.215945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.215989 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.216586 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.217276 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.218377 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.218598 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.218657 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.219445 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.219446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.219446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.219450 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.219459 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.219462 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.219499 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.219583 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.219604 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:09:21.219954 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.215244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42893 (local address 127.0.98.3:59465)
0924 03:09:21.215423 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.215451 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.215849 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.216042 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.216045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.216052 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.216488 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.216492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.217366 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.217367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.218305 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:21.218308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:21.218692 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:21.218693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:21.218815 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:21.219325 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:21.219325 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.219358 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:21.219377 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.219500 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.219501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.219503 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.219573 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.219721 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260924 03:09:21.220430 10077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42893
I20260924 03:09:21.220505 10077 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.220672 10077 heartbeater.cc:511] Master 127.0.98.62:42893 requested a full tablet report, sending...
I20260924 03:09:21.220973 9627 ts_manager.cc:194] Registered new tserver with Master: aac6b683ecc2405da3d12874277174ef (127.0.98.3:45817)
I20260924 03:09:21.221252 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59465
I20260924 03:09:21.228816 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:09:21.230521 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:21.230045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.62:42893)
0924 03:09:21.230114 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.230116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.230216 (+ 100us) server_negotiation.cc:407] Connection header received
0924 03:09:21.230292 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.230293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.230304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.230366 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.230484 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41138: BlockingRecv error: recv got EOF from 127.0.0.1:41138 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:09:21.230656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9687
I20260924 03:09:21.234831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9818
I20260924 03:09:21.238950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9949
I20260924 03:09:21.242637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 9596
2026-09-24T03:09:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T03:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:21Z Disabled control of system clock
I20260924 03:09:21.267263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45643
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:09:21.341689 10088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.341907 10088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.341926 10088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.343117 10088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:21.343158 10088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.343170 10088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.343182 10088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:21.343194 10088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:21.344435 10088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.344627 10088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.344892 10088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.348994 10094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.348958 10093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.348976 10088 server_base.cc:1034] running on GCE node
W20260924 03:09:21.349269 10096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.349524 10088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.349787 10088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.350943 10088 hybrid_clock.cc:648] HybridClock initialized: now 1790219361350930 us; error 25 us; skew 500 ppm
I20260924 03:09:21.352022 10088 webserver.cc:460] Webserver started at http://127.0.98.62:46389/ using document root <none> and password file <none>
I20260924 03:09:21.352267 10088 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.352329 10088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.352454 10088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.353385 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "6e93464ad3c848118b8ea70009d70814"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "35cda9da70c6ebd49b9188e4b251a2ce"
server_key_iv: "45baab29a0866a5a90f38dde2b01c2e2"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.353718 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e93464ad3c848118b8ea70009d70814"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "35cda9da70c6ebd49b9188e4b251a2ce"
server_key_iv: "45baab29a0866a5a90f38dde2b01c2e2"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.354880 10088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:09:21.355507 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.355677 10088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:21.355756 10088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "6e93464ad3c848118b8ea70009d70814"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "35cda9da70c6ebd49b9188e4b251a2ce"
server_key_iv: "45baab29a0866a5a90f38dde2b01c2e2"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.355826 10088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.358729 10088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.358973 10088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.359107 10088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.362779 10088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45643
I20260924 03:09:21.362805 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45643 every 8 connection(s)
I20260924 03:09:21.363085 10088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:21.363626 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:21.365751 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814: Bootstrap starting.
I20260924 03:09:21.366284 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:21.366545 10155 log.cc:824] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:21.367110 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814: No bootstrap required, opened a new log
I20260924 03:09:21.368458 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } }
I20260924 03:09:21.368569 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:21.368589 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e93464ad3c848118b8ea70009d70814, State: Initialized, Role: FOLLOWER
I20260924 03:09:21.368702 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } }
I20260924 03:09:21.368767 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:21.368788 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:21.368815 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:21.369221 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } }
I20260924 03:09:21.369292 10155 leader_election.cc:302] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e93464ad3c848118b8ea70009d70814; no voters:
I20260924 03:09:21.369400 10155 leader_election.cc:288] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:21.369474 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:21.369633 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [term 1 LEADER]: Becoming Leader. State: Replica: 6e93464ad3c848118b8ea70009d70814, State: Running, Role: LEADER
I20260924 03:09:21.369642 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:21.369715 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } }
I20260924 03:09:21.370041 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e93464ad3c848118b8ea70009d70814. Latest consensus state: current_term: 1 leader_uuid: "6e93464ad3c848118b8ea70009d70814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } } }
I20260924 03:09:21.370126 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:21.370391 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e93464ad3c848118b8ea70009d70814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e93464ad3c848118b8ea70009d70814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45643 } } }
I20260924 03:09:21.370461 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:21.370471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10088
I20260924 03:09:21.370568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:09:21.370602 10166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:21.370754 392 external_mini_cluster.cc:1428] Setting key 1fe783f05aecc1feb1bba2ce987b88e4
I20260924 03:09:21.371068 10166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:21.372767 10166 catalog_manager.cc:1347] Generated new cluster ID: e1ef4492607c4ac38a6bed0a278280cc
I20260924 03:09:21.372821 10166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:09:21.373085 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.371332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.98.62:45643)
0924 03:09:21.371520 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.371527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.371556 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:09:21.371705 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.371711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.371768 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.371913 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.372620 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.372628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.372629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.372633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.372664 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.372692 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.372751 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.372910 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.372964 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260924 03:09:21.404367 10166 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:21.404489 10166 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:21.413771 10166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e93464ad3c848118b8ea70009d70814: Generated new TSK 0
I20260924 03:09:21.413908 10166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:09:21.416432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:09:21.490144 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.490386 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.490406 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.491431 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.491463 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.491487 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:21.493198 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.493420 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.493690 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.497885 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.497920 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.497951 10179 server_base.cc:1034] running on GCE node
W20260924 03:09:21.497885 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.498252 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.498458 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.499593 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790219361499577 us; error 33 us; skew 500 ppm
I20260924 03:09:21.500593 10179 webserver.cc:460] Webserver started at http://127.0.98.1:43227/ using document root <none> and password file <none>
I20260924 03:09:21.500804 10179 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.500844 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.500975 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.501896 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "98f9aa341d344c71937a899e6443bcfe"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e9027428d6a40978faf9fdf59e64c3b9"
server_key_iv: "0c790db52abb1b8a6a0853bb55377a18"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.502285 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98f9aa341d344c71937a899e6443bcfe"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e9027428d6a40978faf9fdf59e64c3b9"
server_key_iv: "0c790db52abb1b8a6a0853bb55377a18"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.503456 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:21.504068 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.504245 10179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:21.504323 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "98f9aa341d344c71937a899e6443bcfe"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "e9027428d6a40978faf9fdf59e64c3b9"
server_key_iv: "0c790db52abb1b8a6a0853bb55377a18"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.504390 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.507160 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.507370 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.507540 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.507725 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.507983 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.508013 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.508059 10179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.508083 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.513398 10179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38623
I20260924 03:09:21.513432 10306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38623 every 8 connection(s)
I20260924 03:09:21.513676 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:21.516083 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.514241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38295 (local address 127.0.98.62:45643)
0924 03:09:21.514281 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.514282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.514497 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:09:21.514894 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.514895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.514902 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.514940 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.515883 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.515884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.515884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.515888 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.515902 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.515907 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.515951 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.516008 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.516022 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:09:21.516253 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.514237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45643 (local address 127.0.98.1:38295)
0924 03:09:21.514399 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.514426 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.514804 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.515007 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.515010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.515018 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.515748 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:21.515748 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.515770 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:21.515793 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.515948 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.515952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.515955 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.516018 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.516067 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260924 03:09:21.516673 10307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45643
I20260924 03:09:21.516736 10307 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.516878 10307 heartbeater.cc:511] Master 127.0.98.62:45643 requested a full tablet report, sending...
I20260924 03:09:21.517295 10119 ts_manager.cc:194] Registered new tserver with Master: 98f9aa341d344c71937a899e6443bcfe (127.0.98.1:38623)
I20260924 03:09:21.520370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10179
I20260924 03:09:21.520437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:21.520572 392 external_mini_cluster.cc:1428] Setting key c3285e02fc8e2352d0d3d7dfb44ee993
I20260924 03:09:21.521248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:09:21.594668 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.594878 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.594899 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.595944 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.595974 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.595996 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:21.597105 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.597254 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.597451 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.601778 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.601800 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.601841 10310 server_base.cc:1034] running on GCE node
W20260924 03:09:21.601787 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.602131 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.602388 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.603542 10310 hybrid_clock.cc:648] HybridClock initialized: now 1790219361603532 us; error 40 us; skew 500 ppm
I20260924 03:09:21.604611 10310 webserver.cc:460] Webserver started at http://127.0.98.2:34261/ using document root <none> and password file <none>
I20260924 03:09:21.604827 10310 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.604889 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.605008 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.605898 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "4aa8436db59040b098eae0f2009d9108"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "b7f92195996b9a55db1338a3301fd8de"
server_key_iv: "c7aa52a44c3bef839b5f00b603a0310a"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.606266 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa8436db59040b098eae0f2009d9108"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "b7f92195996b9a55db1338a3301fd8de"
server_key_iv: "c7aa52a44c3bef839b5f00b603a0310a"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.607460 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:09:21.608124 10324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.608268 10310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:21.608372 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "4aa8436db59040b098eae0f2009d9108"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "b7f92195996b9a55db1338a3301fd8de"
server_key_iv: "c7aa52a44c3bef839b5f00b603a0310a"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.608459 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.611805 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.612053 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.612203 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.612416 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.612697 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.612748 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.612787 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.612813 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.618275 10310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43851
I20260924 03:09:21.618324 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43851 every 8 connection(s)
I20260924 03:09:21.618609 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:09:21.621304 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.619282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53225 (local address 127.0.98.62:45643)
0924 03:09:21.619363 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.619366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.619552 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:09:21.620002 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.620004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.620010 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.620071 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.621076 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.621077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.621077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.621082 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.621098 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.621103 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.621149 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.621215 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.621229 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:09:21.621485 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.619253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45643 (local address 127.0.98.2:53225)
0924 03:09:21.619431 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.619475 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.619901 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.620131 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.620135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.620144 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.620944 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:21.620944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.620967 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:21.620991 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.621155 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.621159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.621162 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.621226 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.621278 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260924 03:09:21.621899 10438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45643
I20260924 03:09:21.621981 10438 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.622136 10438 heartbeater.cc:511] Master 127.0.98.62:45643 requested a full tablet report, sending...
I20260924 03:09:21.622485 10119 ts_manager.cc:194] Registered new tserver with Master: 4aa8436db59040b098eae0f2009d9108 (127.0.98.2:43851)
I20260924 03:09:21.625150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10310
I20260924 03:09:21.625218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:21.625362 392 external_mini_cluster.cc:1428] Setting key 9dd30bbfb341b07ff13912891a35f2f4
I20260924 03:09:21.626065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:09:21.700284 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:21.700500 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:21.700536 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:21.701622 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:21.701683 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:21.701766 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:21.703121 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:21.703379 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:21.703635 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:21.708329 10447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.708364 10449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:21.708338 10446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:21.708562 10441 server_base.cc:1034] running on GCE node
I20260924 03:09:21.708732 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:21.708949 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:21.710098 10441 hybrid_clock.cc:648] HybridClock initialized: now 1790219361710083 us; error 26 us; skew 500 ppm
I20260924 03:09:21.711259 10441 webserver.cc:460] Webserver started at http://127.0.98.3:44607/ using document root <none> and password file <none>
I20260924 03:09:21.711479 10441 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:21.711545 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:21.711686 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:21.712760 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "6145d7bf6fbe402e9648458496665aec"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "9388e7e0e3d2881f1bfcda16efd899fa"
server_key_iv: "a2f856feaea1810658aadf2d909a5b27"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.713115 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6145d7bf6fbe402e9648458496665aec"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "9388e7e0e3d2881f1bfcda16efd899fa"
server_key_iv: "a2f856feaea1810658aadf2d909a5b27"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.714356 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:09:21.715024 10455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.715190 10441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:21.715281 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "6145d7bf6fbe402e9648458496665aec"
format_stamp: "Formatted at 2026-09-24 03:09:21 on dist-test-slave-jtnz"
server_key: "9388e7e0e3d2881f1bfcda16efd899fa"
server_key_iv: "a2f856feaea1810658aadf2d909a5b27"
server_key_version: "encryptionkey@0"
I20260924 03:09:21.715359 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:21.718432 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:21.718679 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:21.718809 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:21.719028 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:21.719296 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:21.719345 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.719380 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:21.719409 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:21.724793 10441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43301
I20260924 03:09:21.724823 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43301 every 8 connection(s)
I20260924 03:09:21.725106 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:09:21.727705 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.725728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48163 (local address 127.0.98.62:45643)
0924 03:09:21.725789 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.725791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.726004 (+ 213us) server_negotiation.cc:407] Connection header received
0924 03:09:21.726482 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.726483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.726487 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.726539 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.727515 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:21.727517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:21.727517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:09:21.727521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:21.727533 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:09:21.727537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:21.727587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:21.727646 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:09:21.727657 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:09:21.727903 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:21.725705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45643 (local address 127.0.98.3:48163)
0924 03:09:21.725886 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:21.725923 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:21.726399 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:21.726585 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:21.726588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:21.726596 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:21.727385 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:09:21.727385 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:21.727409 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:09:21.727434 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:21.727595 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:21.727599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:21.727601 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:09:21.727653 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:09:21.727695 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260924 03:09:21.728407 10569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45643
I20260924 03:09:21.728514 10569 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:21.728657 10569 heartbeater.cc:511] Master 127.0.98.62:45643 requested a full tablet report, sending...
I20260924 03:09:21.729004 10119 ts_manager.cc:194] Registered new tserver with Master: 6145d7bf6fbe402e9648458496665aec (127.0.98.3:43301)
I20260924 03:09:21.729976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10441
I20260924 03:09:21.730073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:21.730273 392 external_mini_cluster.cc:1428] Setting key b9a2cdcac9f8a23531d6f03cc5f2b3d0
I20260924 03:09:21.731105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:09:21.732837 10177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:09:21.732393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.0.98.62:45643)
0924 03:09:21.732455 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:21.732457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:21.732525 (+ 68us) server_negotiation.cc:407] Connection header received
0924 03:09:21.732570 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:21.732571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:21.732574 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:21.732617 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:21.732777 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35958: BlockingRecv error: recv got EOF from 127.0.0.1:35958 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:09:21.732870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10179
I20260924 03:09:21.737974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10310
I20260924 03:09:21.742203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10441
I20260924 03:09:21.746434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10088
2026-09-24T03:09:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (504 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:21 dist-test-slave-jtnz krb5kdc[10579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:21 dist-test-slave-jtnz krb5kdc[10579](info): set up 2 sockets
Sep 24 03:09:21 dist-test-slave-jtnz krb5kdc[10579](info): commencing operation
krb5kdc: starting...
W20260924 03:09:22.722831 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:09:22 dist-test-slave-jtnz krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:09:22.826898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:22.902139 10592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:22.902419 10592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:22.902441 10592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:22.903530 10592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:09:22.903563 10592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:22.903582 10592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:22.903594 10592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:09:22.903606 10592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:09:22.904893 10592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:22.905051 10592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:22.905194 10592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:22.909420 10600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:22.909415 10597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:22.909550 10598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:22.909670 10592 server_base.cc:1034] running on GCE node
I20260924 03:09:22.909899 10592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:22.910161 10592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:22.911361 10592 hybrid_clock.cc:648] HybridClock initialized: now 1790219362911331 us; error 44 us; skew 500 ppm
Sep 24 03:09:22 dist-test-slave-jtnz krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:22.913792 10592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:09:22.914260 10592 webserver.cc:460] Webserver started at http://127.0.98.62:46525/ using document root <none> and password file <none>
I20260924 03:09:22.914445 10592 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:22.914494 10592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:22.914592 10592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:22.915557 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "49355801996e4c68b403739c18c3e258"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-jtnz"
server_key: "6e75c4792e9d9b74fdcf8c23310b11ad"
server_key_iv: "b97f1b825b6891a2159c521496b58b66"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.916118 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "49355801996e4c68b403739c18c3e258"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-jtnz"
server_key: "6e75c4792e9d9b74fdcf8c23310b11ad"
server_key_iv: "b97f1b825b6891a2159c521496b58b66"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.917326 10592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:09:22.917940 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:22.918095 10592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:09:22.918169 10592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "49355801996e4c68b403739c18c3e258"
format_stamp: "Formatted at 2026-09-24 03:09:22 on dist-test-slave-jtnz"
server_key: "6e75c4792e9d9b74fdcf8c23310b11ad"
server_key_iv: "b97f1b825b6891a2159c521496b58b66"
server_key_version: "encryptionkey@0"
I20260924 03:09:22.918268 10592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:22.929505 10592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:22.931139 10592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:22.931236 10592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:22.934798 10592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44941
I20260924 03:09:22.934842 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44941 every 8 connection(s)
I20260924 03:09:22.935124 10592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:09:22.936017 10660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:22.938037 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: Bootstrap starting.
I20260924 03:09:22.938637 10660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:22.938906 10660 log.cc:824] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:22.939512 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: No bootstrap required, opened a new log
I20260924 03:09:22.940806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10592
I20260924 03:09:22.940877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:09:22.941026 392 external_mini_cluster.cc:1428] Setting key 445fee5304b7b15ed7e5a6091b213b87
I20260924 03:09:22.941095 10660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } }
I20260924 03:09:22.941241 10660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:22.941298 10660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49355801996e4c68b403739c18c3e258, State: Initialized, Role: FOLLOWER
I20260924 03:09:22.941438 10660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } }
I20260924 03:09:22.941525 10660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:09:22.941588 10660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:09:22.941644 10660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:22.942456 10660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } }
I20260924 03:09:22.942571 10660 leader_election.cc:302] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49355801996e4c68b403739c18c3e258; no voters:
I20260924 03:09:22.942735 10660 leader_election.cc:288] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:09:22.942790 10665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:22.942960 10665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [term 1 LEADER]: Becoming Leader. State: Replica: 49355801996e4c68b403739c18c3e258, State: Running, Role: LEADER
I20260924 03:09:22.943008 10660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:09:22.943106 10665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } }
I20260924 03:09:22.943486 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49355801996e4c68b403739c18c3e258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } } }
I20260924 03:09:22.943557 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [sys.catalog]: This master's current role is: LEADER
I20260924 03:09:22.943746 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49355801996e4c68b403739c18c3e258. Latest consensus state: current_term: 1 leader_uuid: "49355801996e4c68b403739c18c3e258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49355801996e4c68b403739c18c3e258" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44941 } } }
I20260924 03:09:22.944013 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258 [sys.catalog]: This master's current role is: LEADER
W20260924 03:09:22.944801 10681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:09:22.945196 10681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:09:22.945314 10671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:09:22.945519 10671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:09:22.946930 10671 catalog_manager.cc:1347] Generated new cluster ID: 0fa965995f804847853c03523abf6290
I20260924 03:09:22.946977 10671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:09:22 dist-test-slave-jtnz krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:22.952589 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:22.941484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.98.62:44941)
0924 03:09:22.941646 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:22.941654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:22.941682 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:09:22.941806 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:22.941811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:22.941848 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:22.941968 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:22.942797 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.944196 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.944946 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:22.945511 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:22.945620 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:22.948392 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:22.948396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:22.948396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:22.948407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:22.950334 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.950715 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.950718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.950718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.950842 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:22.951045 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:22.951046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:22.951046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:22.951322 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:22.952178 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:22.952263 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:09:22.952399 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260924 03:09:22.962127 10671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:09:22.962625 10671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:09:22.967196 10671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: Generated new TSK 0
I20260924 03:09:22.967344 10671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:09:22.990752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:23.073041 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:23.073217 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:23.073240 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:23.074379 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:23.074415 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:23.074440 10686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:09:23.075722 10686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:23.075892 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:23.076038 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:23.080513 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:23.080524 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:23.080513 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:23.080828 10686 server_base.cc:1034] running on GCE node
I20260924 03:09:23.081013 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:23.081243 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:23.082397 10686 hybrid_clock.cc:648] HybridClock initialized: now 1790219363082401 us; error 43 us; skew 500 ppm
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:23.084765 10686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:09:23.085148 10686 webserver.cc:460] Webserver started at http://127.0.98.1:43479/ using document root <none> and password file <none>
I20260924 03:09:23.085350 10686 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:23.085414 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:23.085548 10686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:23.086474 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "91f0b442ba1947e182e91db12d4369b1"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "7bef7e37ab9a8af9c57dfab0273a0cca"
server_key_iv: "0a8f745251e707403fe7eae0a6695348"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.086788 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91f0b442ba1947e182e91db12d4369b1"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "7bef7e37ab9a8af9c57dfab0273a0cca"
server_key_iv: "0a8f745251e707403fe7eae0a6695348"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.087939 10686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:23.088522 10701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.088687 10686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:23.088768 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "91f0b442ba1947e182e91db12d4369b1"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "7bef7e37ab9a8af9c57dfab0273a0cca"
server_key_iv: "0a8f745251e707403fe7eae0a6695348"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.088841 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:23.103140 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:23.105278 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:23.105424 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:23.105607 10686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:23.105857 10686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:23.105906 10686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.105940 10686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:23.105962 10686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.111325 10686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38655
I20260924 03:09:23.111364 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38655 every 8 connection(s)
I20260924 03:09:23.111660 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:09:23.114738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10686
I20260924 03:09:23.114818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:09:23.114992 392 external_mini_cluster.cc:1428] Setting key 51c5541d81b0a0d3ef57d09a0d1026e0
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:23.121268 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.112323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55797 (local address 127.0.98.62:44941)
0924 03:09:23.112370 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.112372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.112540 (+ 168us) server_negotiation.cc:407] Connection header received
0924 03:09:23.112945 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.112946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.112956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.113012 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:23.113792 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.114329 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.115584 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.115794 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.115854 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.118279 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:23.118280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:23.118281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:23.118288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:23.119922 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.120285 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.120287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.120287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.120359 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.120637 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.120638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.120638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.120747 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:23.120821 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.121089 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.121176 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:09:23.121387 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.112275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44941 (local address 127.0.98.1:55797)
0924 03:09:23.112434 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.112471 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.112865 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.113060 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.113063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.113577 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:23.113725 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.113729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.114399 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.114400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.115503 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.115506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.115874 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.115875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.116016 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.116435 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:23.116444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:23.118132 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:23.120005 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.120011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.120013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.120213 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.120444 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.120445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.120446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.120543 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.120814 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:23.120820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:23.120944 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.121037 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.121179 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260924 03:09:23.121846 10815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44941
I20260924 03:09:23.121951 10815 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:23.122133 10815 heartbeater.cc:511] Master 127.0.98.62:44941 requested a full tablet report, sending...
I20260924 03:09:23.122653 10624 ts_manager.cc:194] Registered new tserver with Master: 91f0b442ba1947e182e91db12d4369b1 (127.0.98.1:38655)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:09:23.123240 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:09:23.136752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:23.210937 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:23.211098 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:23.211118 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:23.212213 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:23.212253 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:23.212296 10820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:09:23.213632 10820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:23.213840 10820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:23.214035 10820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:23.218453 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:23.218489 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:23.218536 10820 server_base.cc:1034] running on GCE node
W20260924 03:09:23.218480 10825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:23.218825 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:23.219110 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:23.220292 10820 hybrid_clock.cc:648] HybridClock initialized: now 1790219363220275 us; error 29 us; skew 500 ppm
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:23.222910 10820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:09:23.223301 10820 webserver.cc:460] Webserver started at http://127.0.98.2:44361/ using document root <none> and password file <none>
I20260924 03:09:23.223522 10820 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:23.223582 10820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:23.223716 10820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:23.224608 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3babfe43ded4375a9d4acc70f0e145c"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "2c73d565979bbe7190d771082b39dfa8"
server_key_iv: "6d9cdc0cb603547d032cf7d9ba7b061f"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.224957 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3babfe43ded4375a9d4acc70f0e145c"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "2c73d565979bbe7190d771082b39dfa8"
server_key_iv: "6d9cdc0cb603547d032cf7d9ba7b061f"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.226176 10820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:09:23.226902 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.227052 10820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:23.227142 10820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "f3babfe43ded4375a9d4acc70f0e145c"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "2c73d565979bbe7190d771082b39dfa8"
server_key_iv: "6d9cdc0cb603547d032cf7d9ba7b061f"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.227222 10820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:23.233791 10820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:23.235450 10820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:23.235603 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:23.235785 10820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:23.236056 10820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:23.236106 10820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.236143 10820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:23.236171 10820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.241305 10820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44367
I20260924 03:09:23.241366 10948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44367 every 8 connection(s)
I20260924 03:09:23.241648 10820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:23.249491 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.242428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51151 (local address 127.0.98.62:44941)
0924 03:09:23.242490 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.242492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.242642 (+ 150us) server_negotiation.cc:407] Connection header received
0924 03:09:23.243096 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.243097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.243106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.243177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:23.244051 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.244600 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.245463 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.245628 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.245672 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.247409 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:23.247410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:23.247410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:23.247416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:23.248474 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.248804 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.248805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.248805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.248849 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.249010 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.249010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.249010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.249110 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:23.249158 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.249376 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.249433 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:09:23.249622 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.242394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44941 (local address 127.0.98.2:51151)
0924 03:09:23.242531 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.242567 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.243006 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.243232 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.243235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.243724 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:23.243983 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.243988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.244667 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.244667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.245397 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.245399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.245688 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.245689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.245771 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.246100 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:23.246106 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:23.247333 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:23.248538 (+ 1205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.248543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.248545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.248734 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.248888 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.248888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.248888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.248964 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.249165 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:23.249169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:23.249272 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.249345 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.249444 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 03:09:23.250042 10949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44941
I20260924 03:09:23.250124 10949 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:23.250299 10949 heartbeater.cc:511] Master 127.0.98.62:44941 requested a full tablet report, sending...
I20260924 03:09:23.250650 10624 ts_manager.cc:194] Registered new tserver with Master: f3babfe43ded4375a9d4acc70f0e145c (127.0.98.2:44367)
I20260924 03:09:23.250765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10820
I20260924 03:09:23.250849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:09:23.251008 392 external_mini_cluster.cc:1428] Setting key 0659ff4fbdb1945bbafd5b220113f582
I20260924 03:09:23.251124 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51151
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:09:23.272425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:09:23.346141 10954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:09:23.346346 10954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:09:23.346377 10954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:09:23.347383 10954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:09:23.347437 10954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:09:23.347491 10954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:09:23.348742 10954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:09:23.348968 10954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:09:23.349155 10954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:09:23.353466 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:09:23.353518 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:23.353595 10954 server_base.cc:1034] running on GCE node
W20260924 03:09:23.353463 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:09:23.353864 10954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:09:23.354064 10954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:09:23.355211 10954 hybrid_clock.cc:648] HybridClock initialized: now 1790219363355202 us; error 25 us; skew 500 ppm
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:23.357254 10954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:09:23.357647 10954 webserver.cc:460] Webserver started at http://127.0.98.3:46711/ using document root <none> and password file <none>
I20260924 03:09:23.357843 10954 fs_manager.cc:359] Metadata directory not provided
I20260924 03:09:23.357900 10954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:09:23.358024 10954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:09:23.359006 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "129aa33d363b4960a73c232098c8e5a6"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "52803dc248164b36bd4b012e3a6a94b7"
server_key_iv: "d209d2058ba5e0b8a13a993c90701967"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.359340 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "129aa33d363b4960a73c232098c8e5a6"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "52803dc248164b36bd4b012e3a6a94b7"
server_key_iv: "d209d2058ba5e0b8a13a993c90701967"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.360532 10954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:09:23.361176 10969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.361332 10954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:09:23.361441 10954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "129aa33d363b4960a73c232098c8e5a6"
format_stamp: "Formatted at 2026-09-24 03:09:23 on dist-test-slave-jtnz"
server_key: "52803dc248164b36bd4b012e3a6a94b7"
server_key_iv: "d209d2058ba5e0b8a13a993c90701967"
server_key_version: "encryptionkey@0"
I20260924 03:09:23.361528 10954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:09:23.382617 10954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:09:23.384262 10954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:09:23.384407 10954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:09:23.384609 10954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:09:23.384884 10954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:09:23.384936 10954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.384975 10954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:09:23.385002 10954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:09:23.390086 10954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42145
I20260924 03:09:23.390132 11082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42145 every 8 connection(s)
I20260924 03:09:23.390407 10954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:09:23.396216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 10954
I20260924 03:09:23.396301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:09:23.396499 392 external_mini_cluster.cc:1428] Setting key 78aa17e8623c611c97612b041040be9d
I20260924 03:09:23.398272 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.391033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56483 (local address 127.0.98.62:44941)
0924 03:09:23.391081 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.391084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.391312 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:09:23.391752 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.391753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.391762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.391810 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:23.392697 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.393189 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.394018 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.394168 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.394254 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.395954 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:23.395955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:23.395955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:23.395961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:23.397276 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.397601 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.397601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.397602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.397643 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.397788 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.397788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.397789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.397876 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:23.397919 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.398116 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.398163 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:09:23.398408 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.391015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44941 (local address 127.0.98.3:56483)
0924 03:09:23.391195 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.391232 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.391672 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.391862 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.391865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.392460 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:09:23.392628 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.392632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.393258 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.393259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.393955 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.393956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.394259 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.394260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.394379 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.394727 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:09:23.394733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:09:23.395884 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:09:23.397342 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.397347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.397348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.397545 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.397679 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:09:23.397679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:09:23.397680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:09:23.397747 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:09:23.397926 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:09:23.397930 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:09:23.398020 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.398089 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.398226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260924 03:09:23.398768 11083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44941
I20260924 03:09:23.398881 11083 heartbeater.cc:464] Registering TS with master...
I20260924 03:09:23.399032 11083 heartbeater.cc:511] Master 127.0.98.62:44941 requested a full tablet report, sending...
I20260924 03:09:23.399433 10624 ts_manager.cc:194] Registered new tserver with Master: 129aa33d363b4960a73c232098c8e5a6 (127.0.98.3:42145)
I20260924 03:09:23.399778 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56483
I20260924 03:09:23.407825 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:09:23.414012 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.408911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.0.98.62:44941)
0924 03:09:23.408960 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.408962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.409087 (+ 125us) server_negotiation.cc:407] Connection header received
0924 03:09:23.409119 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.409120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.409126 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.409169 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:09:23.409629 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.410022 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.410748 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.410918 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.410966 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.411950 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:09:23.411952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:09:23.411952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:09:23.411957 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:09:23.413218 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.413476 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.413476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.413476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.413517 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:09:23.413656 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:09:23.413656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:09:23.413656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:09:23.413753 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:09:23.413813 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.413893 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.413939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:09:23.415313 10624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:09:23.416685 10624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:09:23.422710 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38655 (local address 127.0.0.1:43150)
0924 03:09:23.418625 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.418656 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.418715 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.419248 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.419249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.419251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:23.419302 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.419304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420624 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.420625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.421538 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.421540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422435 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.422507 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422520 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.422587 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.422632 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":31,"thread_start_us":52,"threads_started":1}
I20260924 03:09:23.422734 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.98.1:38655)
0924 03:09:23.419101 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.419105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.419113 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:23.419152 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.419154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.419165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.419250 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:23.419470 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420486 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.421615 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422326 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422424 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422479 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.422603 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.422646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":74,"threads_started":1}
I20260924 03:09:23.423085 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.0.98.3:42145)
0924 03:09:23.418742 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.418746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.419018 (+ 272us) server_negotiation.cc:407] Connection header received
0924 03:09:23.419091 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.419093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.419106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.419191 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:23.419309 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420423 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.421924 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422524 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422595 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422642 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.422985 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.423017 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260924 03:09:23.423080 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42145 (local address 127.0.0.1:60486)
0924 03:09:23.418939 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.418949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.419023 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.419196 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.419199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.419201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:23.419257 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.419258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420530 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.420531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.421834 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.421836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422824 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.422896 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422900 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.422971 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.423012 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":67,"thread_start_us":45,"threads_started":1}
I20260924 03:09:23.423139 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.98.2:44367)
0924 03:09:23.419051 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.419055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.419409 (+ 354us) server_negotiation.cc:407] Connection header received
0924 03:09:23.419482 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.419484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.419499 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.419571 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:23.419764 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420478 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.421898 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422636 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422704 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422755 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.422995 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.423045 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260924 03:09:23.422935 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.418790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44367 (local address 127.0.0.1:51332)
0924 03:09:23.419342 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.419351 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.419408 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.419649 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.419650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.419652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:23.419693 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.419694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.420599 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.420599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.421826 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.421829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.422702 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.422703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.422763 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.422767 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.422826 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.422880 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":104,"thread_start_us":78,"threads_started":1}
I20260924 03:09:23.423725 11017 tablet_service.cc:1467] Processing CreateTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc (DEFAULT_TABLE table=test-table [id=b7d2081a03bc4cf28fc222ff001bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:23.423673 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc (DEFAULT_TABLE table=test-table [id=b7d2081a03bc4cf28fc222ff001bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:23.424047 11017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:23.424052 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:23.424371 10883 tablet_service.cc:1467] Processing CreateTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc (DEFAULT_TABLE table=test-table [id=b7d2081a03bc4cf28fc222ff001bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:09:23.424661 10883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:09:23.426432 11097 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Bootstrap starting.
I20260924 03:09:23.427002 11097 tablet_bootstrap.cc:654] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:23.427201 11098 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: Bootstrap starting.
I20260924 03:09:23.427644 11099 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Bootstrap starting.
I20260924 03:09:23.427762 11097 log.cc:824] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:23.427894 11098 tablet_bootstrap.cc:654] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:23.428253 11099 tablet_bootstrap.cc:654] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Neither blocks nor log segments found. Creating new log.
I20260924 03:09:23.428433 11098 log.cc:824] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:23.428582 11097 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: No bootstrap required, opened a new log
I20260924 03:09:23.428653 11097 ts_tablet_manager.cc:1397] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:23.428736 11099 log.cc:824] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:09:23.429175 11098 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: No bootstrap required, opened a new log
I20260924 03:09:23.429250 11098 ts_tablet_manager.cc:1397] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:23.429476 11099 tablet_bootstrap.cc:492] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: No bootstrap required, opened a new log
I20260924 03:09:23.429546 11099 ts_tablet_manager.cc:1397] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:09:23.430714 11097 raft_consensus.cc:348] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.430909 11097 raft_consensus.cc:374] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:23.430945 11097 raft_consensus.cc:729] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129aa33d363b4960a73c232098c8e5a6, State: Initialized, Role: FOLLOWER
I20260924 03:09:23.430817 11098 raft_consensus.cc:348] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.430994 11098 raft_consensus.cc:374] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:23.431026 11098 raft_consensus.cc:729] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f0b442ba1947e182e91db12d4369b1, State: Initialized, Role: FOLLOWER
I20260924 03:09:23.431072 11097 consensus_queue.cc:260] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.431159 11098 consensus_queue.cc:260] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.431407 11098 ts_tablet_manager.cc:1428] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:09:23.431520 10815 heartbeater.cc:503] Master 127.0.98.62:44941 was elected leader, sending a full tablet report...
I20260924 03:09:23.431469 11099 raft_consensus.cc:348] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.431612 11099 raft_consensus.cc:374] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:09:23.431646 11099 raft_consensus.cc:729] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3babfe43ded4375a9d4acc70f0e145c, State: Initialized, Role: FOLLOWER
I20260924 03:09:23.431661 11083 heartbeater.cc:503] Master 127.0.98.62:44941 was elected leader, sending a full tablet report...
I20260924 03:09:23.431738 11099 consensus_queue.cc:260] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.431978 11099 ts_tablet_manager.cc:1428] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:09:23.432081 10949 heartbeater.cc:503] Master 127.0.98.62:44941 was elected leader, sending a full tablet report...
I20260924 03:09:23.432348 11097 ts_tablet_manager.cc:1428] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260924 03:09:23.492192 10950 tablet.cc:2369] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:09:23.512485 11104 raft_consensus.cc:482] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:09:23.512615 11104 raft_consensus.cc:504] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.513013 11104 leader_election.cc:288] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91f0b442ba1947e182e91db12d4369b1 (127.0.98.1:38655), f3babfe43ded4375a9d4acc70f0e145c (127.0.98.2:44367)
I20260924 03:09:23.516826 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.513251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42121 (local address 127.0.98.2:44367)
0924 03:09:23.513294 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.513297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.513310 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:09:23.513351 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.513351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.513364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.513438 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:23.513635 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.514398 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.515919 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.516462 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.516534 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.516543 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.516705 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.516763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:09:23.516826 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.513259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55947 (local address 127.0.98.1:38655)
0924 03:09:23.513310 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.513313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.513489 (+ 176us) server_negotiation.cc:407] Connection header received
0924 03:09:23.513522 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.513522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.513531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.513571 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:09:23.513723 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.514415 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.515917 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.516461 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.516534 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.516544 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.516702 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.516758 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:09:23.516870 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.513174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44367 (local address 127.0.98.3:42121)
0924 03:09:23.513231 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.513245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.513289 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.513460 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.513461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.513463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:23.513537 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.513539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.514480 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.514481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.515836 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.515838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.516542 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.516543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.516628 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.516633 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.516684 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.516746 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":92}
I20260924 03:09:23.516889 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.513175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38655 (local address 127.0.98.3:55947)
0924 03:09:23.513376 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.513391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:09:23.513468 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:09:23.513591 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:09:23.513591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:09:23.513593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:09:23.513648 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.513649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.514480 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.514481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.515833 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:09:23.515836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.516542 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.516543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:09:23.516628 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.516634 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:09:23.516684 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:09:23.516746 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":139,"thread_start_us":99,"threads_started":1}
I20260924 03:09:23.517220 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7128aa08c9c34ab290fb87cfd7b2ecbc" candidate_uuid: "129aa33d363b4960a73c232098c8e5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f0b442ba1947e182e91db12d4369b1" is_pre_election: true
I20260924 03:09:23.517279 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7128aa08c9c34ab290fb87cfd7b2ecbc" candidate_uuid: "129aa33d363b4960a73c232098c8e5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3babfe43ded4375a9d4acc70f0e145c" is_pre_election: true
I20260924 03:09:23.517366 10769 raft_consensus.cc:2454] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129aa33d363b4960a73c232098c8e5a6 in term 0.
I20260924 03:09:23.517457 10903 raft_consensus.cc:2454] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129aa33d363b4960a73c232098c8e5a6 in term 0.
I20260924 03:09:23.517588 10971 leader_election.cc:302] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129aa33d363b4960a73c232098c8e5a6, 91f0b442ba1947e182e91db12d4369b1; no voters:
I20260924 03:09:23.517724 11104 raft_consensus.cc:2781] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:09:23.517776 11104 raft_consensus.cc:482] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:09:23.517804 11104 raft_consensus.cc:3037] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:23.518466 11104 raft_consensus.cc:504] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.518594 11104 leader_election.cc:288] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [CANDIDATE]: Term 1 election: Requested vote from peers 91f0b442ba1947e182e91db12d4369b1 (127.0.98.1:38655), f3babfe43ded4375a9d4acc70f0e145c (127.0.98.2:44367)
I20260924 03:09:23.518726 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7128aa08c9c34ab290fb87cfd7b2ecbc" candidate_uuid: "129aa33d363b4960a73c232098c8e5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f0b442ba1947e182e91db12d4369b1"
I20260924 03:09:23.518751 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7128aa08c9c34ab290fb87cfd7b2ecbc" candidate_uuid: "129aa33d363b4960a73c232098c8e5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3babfe43ded4375a9d4acc70f0e145c"
I20260924 03:09:23.518801 10769 raft_consensus.cc:3037] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:23.518851 10903 raft_consensus.cc:3037] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:09:23.519474 10769 raft_consensus.cc:2454] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129aa33d363b4960a73c232098c8e5a6 in term 1.
I20260924 03:09:23.519520 10903 raft_consensus.cc:2454] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129aa33d363b4960a73c232098c8e5a6 in term 1.
I20260924 03:09:23.519675 10971 leader_election.cc:302] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129aa33d363b4960a73c232098c8e5a6, 91f0b442ba1947e182e91db12d4369b1; no voters:
I20260924 03:09:23.519783 11104 raft_consensus.cc:2781] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:09:23.519922 11104 raft_consensus.cc:686] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 129aa33d363b4960a73c232098c8e5a6, State: Running, Role: LEADER
I20260924 03:09:23.520041 11104 consensus_queue.cc:237] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } }
I20260924 03:09:23.520960 10624 catalog_manager.cc:5495] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 129aa33d363b4960a73c232098c8e5a6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "129aa33d363b4960a73c232098c8e5a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129aa33d363b4960a73c232098c8e5a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } health_report { overall_health: UNKNOWN } } }
I20260924 03:09:23.537165 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.534951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.3:42145)
0924 03:09:23.534998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:09:23.535000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:09:23.535008 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:09:23.535080 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:09:23.535081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:09:23.535103 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:09:23.535173 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:09:23.535299 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.535717 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.536441 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:09:23.536624 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:09:23.536703 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:09:23.536883 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:09:23.536968 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:09:23.537044 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:09:23.537067 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 03:09:23.537085 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:09:23.538654 10769 raft_consensus.cc:1261] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 91f0b442ba1947e182e91db12d4369b1 [term 1 FOLLOWER]: Refusing update from remote peer 129aa33d363b4960a73c232098c8e5a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:23.538654 10903 raft_consensus.cc:1261] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 1 FOLLOWER]: Refusing update from remote peer 129aa33d363b4960a73c232098c8e5a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:09:23.539011 11107 consensus_queue.cc:1035] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3babfe43ded4375a9d4acc70f0e145c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:23.539124 11104 consensus_queue.cc:1035] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f0b442ba1947e182e91db12d4369b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:09:23.541337 11112 mvcc.cc:205] Tried to move back new op lower bound from 7332738513051934720 to 7332738512978972672. Current Snapshot: MvccSnapshot[applied={T|T < 7332738513051934720}]
I20260924 03:09:23.542162 11111 mvcc.cc:205] Tried to move back new op lower bound from 7332738513051934720 to 7332738512978972672. Current Snapshot: MvccSnapshot[applied={T|T < 7332738513051934720}]
I20260924 03:09:23.543517 10623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:23.543608 10623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:09:23.544283 10623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49355801996e4c68b403739c18c3e258: Sending DeleteTablet for 3 replicas of tablet 7128aa08c9c34ab290fb87cfd7b2ecbc
I20260924 03:09:23.544606 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43150
I20260924 03:09:23.544670 10883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51332
I20260924 03:09:23.544813 11017 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:09:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60486
I20260924 03:09:23.544873 11120 tablet_replica.cc:330] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: stopping tablet replica
I20260924 03:09:23.544986 11120 raft_consensus.cc:2229] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:09:23.545094 11120 raft_consensus.cc:2258] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:23.545105 11121 tablet_replica.cc:330] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: stopping tablet replica
I20260924 03:09:23.545274 11121 raft_consensus.cc:2229] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:09:23.545318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10686
I20260924 03:09:23.546211 11120 ts_tablet_manager.cc:1905] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:23.547850 11121 raft_consensus.cc:2258] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:09:23.548159 11120 ts_tablet_manager.cc:1918] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:23.548234 11120 log.cc:1196] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/7128aa08c9c34ab290fb87cfd7b2ecbc
I20260924 03:09:23.548448 11121 ts_tablet_manager.cc:1905] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:09:23.548518 11120 ts_tablet_manager.cc:1939] T 7128aa08c9c34ab290fb87cfd7b2ecbc P f3babfe43ded4375a9d4acc70f0e145c: Deleting consensus metadata
I20260924 03:09:23.548892 10610 catalog_manager.cc:4841] TS f3babfe43ded4375a9d4acc70f0e145c (127.0.98.2:44367): tablet 7128aa08c9c34ab290fb87cfd7b2ecbc (table test-table [id=b7d2081a03bc4cf28fc222ff001bbf12]) successfully deleted
I20260924 03:09:23.550123 11121 ts_tablet_manager.cc:1918] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:09:23.550277 11121 log.cc:1196] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/7128aa08c9c34ab290fb87cfd7b2ecbc
I20260924 03:09:23.550616 11121 ts_tablet_manager.cc:1939] T 7128aa08c9c34ab290fb87cfd7b2ecbc P 129aa33d363b4960a73c232098c8e5a6: Deleting consensus metadata
W20260924 03:09:23.550912 10609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38655 (error 108)
I20260924 03:09:23.551149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10820
I20260924 03:09:23.551172 10610 catalog_manager.cc:4841] TS 129aa33d363b4960a73c232098c8e5a6 (127.0.98.3:42145): tablet 7128aa08c9c34ab290fb87cfd7b2ecbc (table test-table [id=b7d2081a03bc4cf28fc222ff001bbf12]) successfully deleted
I20260924 03:09:23.551672 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:09:23.551213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38655 (local address 127.0.0.1:43162)
0924 03:09:23.551599 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 03:09:23.551638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367}
W20260924 03:09:23.551813 10609 catalog_manager.cc:4584] TS 91f0b442ba1947e182e91db12d4369b1 (127.0.98.1:38655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7128aa08c9c34ab290fb87cfd7b2ecbc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38655: connect: Connection refused (error 111)
I20260924 03:09:23.555500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10954
I20260924 03:09:23.559875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 10592
2026-09-24T03:09:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1814 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[11125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[11125](info): set up 2 sockets
Sep 24 03:09:23 dist-test-slave-jtnz krb5kdc[11125](info): commencing operation
krb5kdc: starting...
W20260924 03:09:24.532682 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:09:24 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:09:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 03:09:24.674901 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskapYcIP/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/postgres -l logfile start
2026-09-24 03:09:26.886 UTC [11155] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:09:26.886 UTC [11155] LOG: listening on IPv4 address "127.0.98.20", port 54973
2026-09-24 03:09:26.890 UTC [11155] LOG: listening on Unix socket "/tmp/.s.PGSQL.54973"
2026-09-24 03:09:26.912 UTC [11158] LOG: database system was shut down at 2026-09-24 03:09:25 UTC
2026-09-24 03:09:26.915 UTC [11155] LOG: database system is ready to accept connections
I20260924 03:09:27.842890 392 mini_postgres.cc:96] Postgres bound to 54973
2026-09-24 03:09:27.848 UTC [11166] FATAL: database "slave" does not exist
127.0.98.20:54973 - accepting connections
I20260924 03:09:27.849052 392 mini_ranger.cc:161] Starting Ranger...
I20260924 03:09:27.862434 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:09:28.282577 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:09:28.282657 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin
2026-09-24 03:09:28,560 [I] DB FLAVOR :POSTGRES
2026-09-24 03:09:28,560 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:09:28,560 [I] Checking connection..
2026-09-24 03:09:28,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:09:28,793 [I] Checking connection passed.
2026-09-24 03:09:28,793 [I] --------- Verifying version history table ---------
2026-09-24 03:09:28,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:09:29,031 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:09:29,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:09:29,272 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:09:29,272 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:09:29,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:09:29.485 UTC [11230] WARNING: there is no transaction in progress
2026-09-24 03:09:29,490 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:09:29,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:09:29,719 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:09:29,719 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:09:29,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:09:29,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:09:30,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jtnz.c.gcp-upstream.internal','N') ;"
2026-09-24 03:09:30,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:09:30,637 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:09:30,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:09:30,848 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:09:30,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:09:31,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:09:31,328 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:09:31,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:09:32.234 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.247 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.262 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.276 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.291 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.306 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.357 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.363 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.377 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.386 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.396 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.409 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.418 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.423 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.429 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32.435 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 03:09:32,991 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:09:32,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:09:33,229 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:09:33,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:09:33,466 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:09:33,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:09:33,713 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:09:33,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:09:33,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:09:34,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jtnz.c.gcp-upstream.internal';"
2026-09-24 03:09:34,375 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:09:34,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:09:34,609 [I] DB_PATCHES have already been applied
I20260924 03:09:34.615163 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskapYcIP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:09:34.615228 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 03:09:34.618104 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57499
Sep 24, 2026 3:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:09:35.692384 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.843s user 0.000s sys 0.008s
Sep 24, 2026 3:09:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:09:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 3:09:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:09:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:09:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:09:54.869391 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:09:54 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:09:54.939777 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:09:54.947307 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:09:55.665556 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:09:55.665637 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms
I20260924 03:09:55.665658 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54973
2026-09-24 03:09:55,831 [I] DB FLAVOR :POSTGRES
2026-09-24 03:09:55,831 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:09:55,831 [I] Checking connection
2026-09-24 03:09:55,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:09:56,050 [I] connection success
2026-09-24 03:09:56,051 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:09:56,051 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:09:56,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:09:56,273 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:09:56,273 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:09:56,273 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:09:56,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54973/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:09:56,531 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:09:56.912799 392 mini_ranger_kms.cc:310] Created kms service
I20260924 03:09:57.073678 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:09:57.114647 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:09:57 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:09:57.471510 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:09:57.471624 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:09:57.471647 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 03:09:57.474926 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11518
I20260924 03:09:57.475044 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60665
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 03:09:59 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:09:59.565069 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.625s user 0.001s sys 0.148s
I20260924 03:09:59.565218 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:09:59.565291 392 mini_ranger_kms.cc:401] 127.0.98.20:60665/kms/v1/keys
Sep 24 03:09:59 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:10:00 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219399, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:10:02 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:10:02.291045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42943
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60665/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:02.367744 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:02.367955 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:02.367988 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:02.369045 11569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:02.369105 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:02.369127 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:02.369146 11569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:02.369195 11569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:02.370615 11569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60665/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42943
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:02.370888 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:02.371080 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:02.376591 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:02.376570 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:02.376649 11569 server_base.cc:1034] running on GCE node
W20260924 03:10:02.376737 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:02.377055 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:02.377353 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:02.378504 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790219402378479 us; error 37 us; skew 500 ppm
Sep 24 03:10:02 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:02.381078 11569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:02.381580 11569 webserver.cc:460] Webserver started at http://127.0.98.62:38357/ using document root <none> and password file <none>
I20260924 03:10:02.381846 11569 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:02.381901 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:02.382023 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:02 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:02.434551 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "c790ab87ce81477a9036f76c539c59bf"
format_stamp: "Formatted at 2026-09-24 03:10:02 on dist-test-slave-jtnz"
server_key: "187c87fe8868bddfe3b4b68dfc55d380"
server_key_iv: "63875f129a16c14c773defb07472c8aa"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:02.434973 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "c790ab87ce81477a9036f76c539c59bf"
format_stamp: "Formatted at 2026-09-24 03:10:02 on dist-test-slave-jtnz"
server_key: "187c87fe8868bddfe3b4b68dfc55d380"
server_key_iv: "63875f129a16c14c773defb07472c8aa"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:02.436573 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:10:02.448939 11584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:02.449085 11569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:02.449160 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "c790ab87ce81477a9036f76c539c59bf"
format_stamp: "Formatted at 2026-09-24 03:10:02 on dist-test-slave-jtnz"
server_key: "187c87fe8868bddfe3b4b68dfc55d380"
server_key_iv: "63875f129a16c14c773defb07472c8aa"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:02.449267 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:02.477770 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:02.479642 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:02.479784 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:02.484863 11569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42943
I20260924 03:10:02.484928 11636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42943 every 8 connection(s)
I20260924 03:10:02.485251 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:02.485895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 11569
I20260924 03:10:02.485981 11637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:02.485998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:10:02.486219 392 external_mini_cluster.cc:1428] Setting key 3256add4a24297f5c99e9ca7d67ff9aa
I20260924 03:10:02.489461 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: Bootstrap starting.
I20260924 03:10:02.490548 11637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:02.490819 11637 log.cc:824] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:02.491537 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: No bootstrap required, opened a new log
I20260924 03:10:02.493626 11637 raft_consensus.cc:348] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } }
I20260924 03:10:02.493808 11637 raft_consensus.cc:374] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:02.493846 11637 raft_consensus.cc:729] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c790ab87ce81477a9036f76c539c59bf, State: Initialized, Role: FOLLOWER
I20260924 03:10:02.494001 11637 consensus_queue.cc:260] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } }
Sep 24 03:10:02 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:02.494081 11637 raft_consensus.cc:388] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:02.494115 11637 raft_consensus.cc:482] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:02.494172 11637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:02.494822 11637 raft_consensus.cc:504] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } }
I20260924 03:10:02.494937 11637 leader_election.cc:302] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c790ab87ce81477a9036f76c539c59bf; no voters:
I20260924 03:10:02.495076 11637 leader_election.cc:288] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:02.495337 11637 sys_catalog.cc:564] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:02.495317 11642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:02.495836 11642 raft_consensus.cc:686] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [term 1 LEADER]: Becoming Leader. State: Replica: c790ab87ce81477a9036f76c539c59bf, State: Running, Role: LEADER
I20260924 03:10:02.496009 11642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } }
I20260924 03:10:02.496393 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c790ab87ce81477a9036f76c539c59bf. Latest consensus state: current_term: 1 leader_uuid: "c790ab87ce81477a9036f76c539c59bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } } }
I20260924 03:10:02.496778 11637 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:10:02.496569 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:02.497141 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c790ab87ce81477a9036f76c539c59bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c790ab87ce81477a9036f76c539c59bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42943 } } }
I20260924 03:10:02.497232 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:02.499982 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:02.486739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.62:42943)
0924 03:10:02.486938 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:02.486948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:02.486981 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:10:02.487518 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:02.487528 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:02.487597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:02.487749 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:02.488516 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.489830 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.491135 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:02.491708 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:02.491811 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:02.494658 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:02.494662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:02.494663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:02.494673 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:02.496560 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.496961 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.496964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.496964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.497054 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:02.497314 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:02.497315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:02.497315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:02.498897 (+ 1582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:02.499259 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:02.499389 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 03:10:02.499701 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260924 03:10:02.506313 11656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:02.506841 11656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:02.510607 11656 catalog_manager.cc:1347] Generated new cluster ID: 547f74bf801a4d56bfb3c8f7d5159e2f
I20260924 03:10:02.510710 11656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:02.518050 11656 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:02.518798 11656 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:02.534232 11656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: Generated new TSK 0
I20260924 03:10:02.534448 11656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:10:03.184657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:03.290954 11682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.291185 11682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.291216 11682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.292707 11682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.292758 11682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.292793 11682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:03.294265 11682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:03.294464 11682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:03.294729 11682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:03.300652 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.300688 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.300747 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.300992 11682 server_base.cc:1034] running on GCE node
I20260924 03:10:03.301191 11682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:03.301469 11682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:03.302649 11682 hybrid_clock.cc:648] HybridClock initialized: now 1790219403302637 us; error 37 us; skew 500 ppm
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:03.305543 11682 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:03.305991 11682 webserver.cc:460] Webserver started at http://127.0.98.1:45283/ using document root <none> and password file <none>
I20260924 03:10:03.306252 11682 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:03.306313 11682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:03.306450 11682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:03.319653 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "52d1aabbe4814a8bba1f1ba487d1282d"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "499a83cc27b68b614ab2b35e68f6c063"
server_key_iv: "6583b3c97499aaa543a5cfd98326c609"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.320068 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52d1aabbe4814a8bba1f1ba487d1282d"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "499a83cc27b68b614ab2b35e68f6c063"
server_key_iv: "6583b3c97499aaa543a5cfd98326c609"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.321792 11682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 03:10:03.334275 11698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.334587 11682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:03.334753 11682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "52d1aabbe4814a8bba1f1ba487d1282d"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "499a83cc27b68b614ab2b35e68f6c063"
server_key_iv: "6583b3c97499aaa543a5cfd98326c609"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.334918 11682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:03.363608 11682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:03.365743 11682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:03.365897 11682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:03.366142 11682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:03.366508 11682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:03.366590 11682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.366652 11682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:03.366694 11682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.373337 11682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44247
I20260924 03:10:03.373804 11682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:03.378880 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44247 every 8 connection(s)
I20260924 03:10:03.380577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 11682
I20260924 03:10:03.380663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:03.380831 392 external_mini_cluster.cc:1428] Setting key 63b0a9e60d9ca14b6098997442dcea49
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:03.392405 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.379588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42943 (local address 127.0.98.1:42391)
0924 03:10:03.379834 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.379869 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.381042 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.381426 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.381431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.383168 (+ 1737us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.383455 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.383460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.384476 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.384478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.385692 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.385694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.386213 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.386214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.386345 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.386934 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.386945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.388649 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:03.390757 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.390763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.390765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.391032 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.391265 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.391265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.391266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.391397 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.391723 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:03.391730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:03.391915 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.392002 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.392148 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
I20260924 03:10:03.393146 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.379582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42391 (local address 127.0.98.62:42943)
0924 03:10:03.381231 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.381233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.381240 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:03.381260 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.381260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.381271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.383550 (+ 2279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.383608 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.384375 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.385788 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.386101 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.388737 (+ 2636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.388783 (+ 46us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:03.388784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:03.388785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:03.388789 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:03.390668 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.391118 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.391119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.391119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.391196 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.391480 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.391480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.391481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.391637 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:03.392906 (+ 1269us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.392957 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.393031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":81,"threads_started":1}
I20260924 03:10:03.393700 11813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42943
I20260924 03:10:03.393778 11813 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:03.393942 11813 heartbeater.cc:511] Master 127.0.98.62:42943 requested a full tablet report, sending...
I20260924 03:10:03.394531 11601 ts_manager.cc:194] Registered new tserver with Master: 52d1aabbe4814a8bba1f1ba487d1282d (127.0.98.1:44247)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:10:03.395337 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:10:03.425604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:10:03.549827 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.550143 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.550253 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.551920 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.552018 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.552098 11819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:03.554299 11819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:03.554738 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:03.555028 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:03.561651 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.562798 11819 server_base.cc:1034] running on GCE node
W20260924 03:10:03.563023 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.563339 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.563625 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:03.563874 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:03.571225 11819 hybrid_clock.cc:648] HybridClock initialized: now 1790219403571202 us; error 35 us; skew 500 ppm
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:03.574436 11819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:03.574895 11819 webserver.cc:460] Webserver started at http://127.0.98.2:39887/ using document root <none> and password file <none>
I20260924 03:10:03.575134 11819 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:03.575193 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:03.575320 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:03.595453 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "68ac7ff126344a7f96c24ddc770d2221"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "e431e4224e76950df974fb05ce35ad56"
server_key_iv: "ba9ce9641d2037547e84c1a9e6ac34a1"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.595955 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68ac7ff126344a7f96c24ddc770d2221"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "e431e4224e76950df974fb05ce35ad56"
server_key_iv: "ba9ce9641d2037547e84c1a9e6ac34a1"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.598760 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:10:03.611075 11836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.611755 11819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:03.611853 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "68ac7ff126344a7f96c24ddc770d2221"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "e431e4224e76950df974fb05ce35ad56"
server_key_iv: "ba9ce9641d2037547e84c1a9e6ac34a1"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.611932 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:03.644419 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:03.649441 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:03.649639 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:03.649925 11819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:03.651226 11819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:03.651508 11819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.651585 11819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:03.651638 11819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.659262 11819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35145
I20260924 03:10:03.659709 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:03.663691 11950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35145 every 8 connection(s)
I20260924 03:10:03.669412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 11819
I20260924 03:10:03.669519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:03.669694 392 external_mini_cluster.cc:1428] Setting key ce1bce08645cbf27d35ed12fe41f877c
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:03.676764 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.663603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60411 (local address 127.0.98.62:42943)
0924 03:10:03.663657 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.663659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.666424 (+ 2765us) server_negotiation.cc:407] Connection header received
0924 03:10:03.667104 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.667105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.667117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.667148 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.667918 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.668743 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.670320 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.670606 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.670648 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.673493 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:03.673494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:03.673495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:03.673501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:03.675251 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.675703 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.675704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.675704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.675757 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.676021 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.676022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.676022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.676164 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:03.676203 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.676551 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.676666 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:10:03.677275 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.662926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42943 (local address 127.0.98.2:60411)
0924 03:10:03.666299 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.666343 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.666973 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.667319 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.667323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.667548 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.667805 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.667822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.668853 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.668853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.670230 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.670232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.670728 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.670728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.670872 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.671865 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.671874 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.673380 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:03.675344 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.675349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.675351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.675625 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.675817 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.675818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.675818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.675945 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.676245 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:03.676250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:03.676404 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.676899 (+ 495us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.677059 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3331,"thread_start_us":494,"threads_started":1}
I20260924 03:10:03.677747 11951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42943
I20260924 03:10:03.677858 11951 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:03.678030 11951 heartbeater.cc:511] Master 127.0.98.62:42943 requested a full tablet report, sending...
I20260924 03:10:03.678406 11601 ts_manager.cc:194] Registered new tserver with Master: 68ac7ff126344a7f96c24ddc770d2221 (127.0.98.2:35145)
I20260924 03:10:03.678902 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60411
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:10:03.707160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:03.823160 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:03.823376 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:03.823450 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:03.824551 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:03.824622 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:03.824666 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:03.826041 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:03.826294 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:03.826505 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:03.831195 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.831431 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:03.831473 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:03.831854 11956 server_base.cc:1034] running on GCE node
I20260924 03:10:03.832041 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:03.832276 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:03.833439 11956 hybrid_clock.cc:648] HybridClock initialized: now 1790219403833432 us; error 29 us; skew 500 ppm
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:03.835949 11956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:03.836397 11956 webserver.cc:460] Webserver started at http://127.0.98.3:40107/ using document root <none> and password file <none>
I20260924 03:10:03.836629 11956 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:03.836695 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:03.836824 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:03.859272 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "7792c1539cd04d83b1d5272db5bf7fc2"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "67316b98d448e39ca9005304df2ab388"
server_key_iv: "ee8f0b45c4ac2c653925bb8ad16f7a9b"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.859829 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7792c1539cd04d83b1d5272db5bf7fc2"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "67316b98d448e39ca9005304df2ab388"
server_key_iv: "ee8f0b45c4ac2c653925bb8ad16f7a9b"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.862473 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:03.879065 11971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.880188 11956 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260924 03:10:03.880272 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "7792c1539cd04d83b1d5272db5bf7fc2"
format_stamp: "Formatted at 2026-09-24 03:10:03 on dist-test-slave-jtnz"
server_key: "67316b98d448e39ca9005304df2ab388"
server_key_iv: "ee8f0b45c4ac2c653925bb8ad16f7a9b"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:03.880337 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:03.900281 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:03.902464 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:03.902606 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:03.902784 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:03.903110 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:03.903170 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.903203 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:03.903220 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:03.912328 12084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46611 every 8 connection(s)
I20260924 03:10:03.912416 11956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46611
I20260924 03:10:03.912798 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:10:03 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:03.920012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 11956
I20260924 03:10:03.920122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:03.920305 392 external_mini_cluster.cc:1428] Setting key 4d1b41b2fe62c9b6832a792ef50099a2
I20260924 03:10:03.923344 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.913604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53681 (local address 127.0.98.62:42943)
0924 03:10:03.913650 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.913652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.913959 (+ 307us) server_negotiation.cc:407] Connection header received
0924 03:10:03.914633 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.914635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.914646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.914675 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.915395 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.916072 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.917432 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.917722 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.917769 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.920187 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:03.920189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:03.920189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:03.920197 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:03.921905 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.922426 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.922427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.922428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.922486 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.922759 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.922760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.922761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.922875 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:03.922937 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.923193 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.923265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:10:03.923466 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.913609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42943 (local address 127.0.98.3:53681)
0924 03:10:03.913840 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.913878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.914487 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.914851 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.914855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.915056 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:03.915289 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.915295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.916174 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.916175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.917333 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.917336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.917863 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.917864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.917981 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.918477 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:03.918486 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:03.920069 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:03.922003 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.922011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.922013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.922339 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.922561 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:03.922562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:03.922563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:03.922671 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:03.922940 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:03.922946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:03.923074 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.923164 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.923278 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260924 03:10:03.923810 12085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42943
I20260924 03:10:03.923883 12085 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:03.924029 12085 heartbeater.cc:511] Master 127.0.98.62:42943 requested a full tablet report, sending...
I20260924 03:10:03.924388 11601 ts_manager.cc:194] Registered new tserver with Master: 7792c1539cd04d83b1d5272db5bf7fc2 (127.0.98.3:46611)
I20260924 03:10:03.924739 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53681
I20260924 03:10:03.932952 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:10:03.941780 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.934420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.0.98.62:42943)
0924 03:10:03.934457 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.934459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.934611 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:10:03.934693 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.934693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.934703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.934730 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:03.935470 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.935938 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.937398 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.937720 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.937773 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.939231 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:03.939232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:03.939232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:03.939240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:03.940641 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.940971 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.940972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.940972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.941025 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:03.941207 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:03.941208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:03.941208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:03.941319 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:03.941403 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.941622 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.941693 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:10:03.943858 11601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:10:03.945567 11601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:10:03.958000 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.949940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35145 (local address 127.0.0.1:45330)
0924 03:10:03.950370 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.950407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.950463 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.950886 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.950888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.950889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:03.950951 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.950952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.951766 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.951767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.953014 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.953016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.957672 (+ 4656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.957686 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.957780 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.957798 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.957869 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.957925 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":74,"threads_started":1}
I20260924 03:10:03.958442 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.950207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.0.98.2:35145)
0924 03:10:03.950588 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.950592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.950600 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:10:03.950736 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.950739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.950753 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.950821 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:03.951052 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.951680 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.953118 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.953941 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.954004 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.955025 (+ 1021us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.958278 (+ 3253us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.958361 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":69,"threads_started":1}
I20260924 03:10:03.958001 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.949959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.0.1:32800)
0924 03:10:03.951817 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.951826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.951865 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.952162 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.952163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.952164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:03.952207 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.952208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.953685 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.953686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.955107 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.955109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.957800 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.957801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.957870 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.957873 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.957912 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.957941 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1839,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260924 03:10:03.959301 11872 tablet_service.cc:1467] Processing CreateTablet for tablet cc4de175d3ab4490b9b8b5a445688355 (DEFAULT_TABLE table=test-table [id=69e7b8c4f96f44febcb3f76eeeca6607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:03.959640 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4de175d3ab4490b9b8b5a445688355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:03.962146 12103 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Bootstrap starting.
I20260924 03:10:03.963178 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.950366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.0.98.1:44247)
0924 03:10:03.951960 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.951964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.951973 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:03.952002 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.952004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.952018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.952102 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:03.952280 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.953599 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.955191 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.956396 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.956467 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.956516 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.963036 (+ 6520us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.963090 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":53,"threads_started":1}
I20260924 03:10:03.963241 12103 tablet_bootstrap.cc:654] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:03.963528 12103 log.cc:824] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:03.963924 11747 tablet_service.cc:1467] Processing CreateTablet for tablet cc4de175d3ab4490b9b8b5a445688355 (DEFAULT_TABLE table=test-table [id=69e7b8c4f96f44febcb3f76eeeca6607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:03.964239 12103 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: No bootstrap required, opened a new log
I20260924 03:10:03.964236 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4de175d3ab4490b9b8b5a445688355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:03.964293 12103 ts_tablet_manager.cc:1397] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:03.966135 12103 raft_consensus.cc:348] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.966302 12103 raft_consensus.cc:374] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:03.966341 12103 raft_consensus.cc:729] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ac7ff126344a7f96c24ddc770d2221, State: Initialized, Role: FOLLOWER
I20260924 03:10:03.966444 12103 consensus_queue.cc:260] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.966707 12103 ts_tablet_manager.cc:1428] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:10:03.968222 11951 heartbeater.cc:503] Master 127.0.98.62:42943 was elected leader, sending a full tablet report...
I20260924 03:10:03.969904 12106 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Bootstrap starting.
I20260924 03:10:03.970587 12106 tablet_bootstrap.cc:654] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:03.970903 12106 log.cc:824] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:03.970932 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.950152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46611 (local address 127.0.0.1:43758)
0924 03:10:03.956763 (+ 6611us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:03.956772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:03.956824 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:03.957080 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:03.957081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:03.957083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:03.957119 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.957120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.968718 (+ 11598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.968719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.969726 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:03.969729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.970739 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.970740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:03.970800 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.970805 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:10:03.970832 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 03:10:03.970867 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6592,"thread_start_us":477,"threads_started":1}
I20260924 03:10:03.971297 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:03.950495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.0.98.3:46611)
0924 03:10:03.956707 (+ 6212us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:03.956711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:03.956880 (+ 169us) server_negotiation.cc:407] Connection header received
0924 03:10:03.956918 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:03.956920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:03.956935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:03.957002 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:03.957196 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.968126 (+ 10930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.969849 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:03.970642 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:03.971091 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:03.971169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:03.971193 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:10:03.971226 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6177,"thread_start_us":78,"threads_started":1}
I20260924 03:10:03.971793 12106 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: No bootstrap required, opened a new log
I20260924 03:10:03.971850 12106 ts_tablet_manager.cc:1397] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:10:03.971858 12019 tablet_service.cc:1467] Processing CreateTablet for tablet cc4de175d3ab4490b9b8b5a445688355 (DEFAULT_TABLE table=test-table [id=69e7b8c4f96f44febcb3f76eeeca6607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:03.972193 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4de175d3ab4490b9b8b5a445688355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:03.974040 12106 raft_consensus.cc:348] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.974191 12106 raft_consensus.cc:374] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:03.974225 12106 raft_consensus.cc:729] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d1aabbe4814a8bba1f1ba487d1282d, State: Initialized, Role: FOLLOWER
I20260924 03:10:03.974349 12106 consensus_queue.cc:260] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.974589 12106 ts_tablet_manager.cc:1428] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:10:03.974846 11813 heartbeater.cc:503] Master 127.0.98.62:42943 was elected leader, sending a full tablet report...
I20260924 03:10:03.975742 12109 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Bootstrap starting.
I20260924 03:10:03.976686 12109 tablet_bootstrap.cc:654] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:03.976996 12109 log.cc:824] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:03.977808 12109 tablet_bootstrap.cc:492] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: No bootstrap required, opened a new log
I20260924 03:10:03.977882 12109 ts_tablet_manager.cc:1397] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:03.980104 12109 raft_consensus.cc:348] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.980257 12109 raft_consensus.cc:374] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:03.980285 12109 raft_consensus.cc:729] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792c1539cd04d83b1d5272db5bf7fc2, State: Initialized, Role: FOLLOWER
I20260924 03:10:03.980387 12109 consensus_queue.cc:260] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:03.980608 12109 ts_tablet_manager.cc:1428] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:10:03.980640 12085 heartbeater.cc:503] Master 127.0.98.62:42943 was elected leader, sending a full tablet report...
Sep 24 03:10:04 dist-test-slave-jtnz krb5kdc[11125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 03:10:04.129587 11814 tablet.cc:2369] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:04.162787 11952 tablet.cc:2369] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:04.163383 12086 tablet.cc:2369] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:10:04.218482 12105 raft_consensus.cc:482] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:04.218647 12105 raft_consensus.cc:504] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:04.219034 12105 leader_election.cc:288] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52d1aabbe4814a8bba1f1ba487d1282d (127.0.98.1:44247), 7792c1539cd04d83b1d5272db5bf7fc2 (127.0.98.3:46611)
I20260924 03:10:04.223242 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.219489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36493 (local address 127.0.98.1:44247)
0924 03:10:04.219544 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.219546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.219783 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:10:04.219853 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.219855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.219867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.219952 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:04.220147 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.220832 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.222340 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.222901 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.222946 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.222955 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:04.223143 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 03:10:04.223180 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:10:04.223670 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.219282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.2:36493)
0924 03:10:04.219725 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.219734 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.219793 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.219983 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.219984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.219987 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:04.220065 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.220067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.220925 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.220926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.222220 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.222223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.223007 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.223008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.223069 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.223073 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:10:04.223535 (+ 462us) client_negotiation.cc:240] Negotiation successful
0924 03:10:04.223588 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260924 03:10:04.224083 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4de175d3ab4490b9b8b5a445688355" candidate_uuid: "68ac7ff126344a7f96c24ddc770d2221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" is_pre_election: true
I20260924 03:10:04.224236 11763 raft_consensus.cc:2454] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ac7ff126344a7f96c24ddc770d2221 in term 0.
I20260924 03:10:04.224457 11837 leader_election.cc:302] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d1aabbe4814a8bba1f1ba487d1282d, 68ac7ff126344a7f96c24ddc770d2221; no voters:
I20260924 03:10:04.224581 12105 raft_consensus.cc:2781] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:04.224619 12105 raft_consensus.cc:482] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:04.224639 12105 raft_consensus.cc:3037] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:04.224668 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.219716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60415 (local address 127.0.98.3:46611)
0924 03:10:04.219761 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.219763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.219970 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:10:04.219999 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.220000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.220010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.220067 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:04.220227 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.220860 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.223630 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.224105 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.224139 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.224146 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:04.224558 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 03:10:04.224601 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:10:04.225346 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.219558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46611 (local address 127.0.98.2:60415)
0924 03:10:04.219878 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.219888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:04.219949 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:04.220111 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:04.220112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:04.220113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:04.220158 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.220159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.220964 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.220964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.222960 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:04.222962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.224397 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.224398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:04.224463 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.224468 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:10:04.225228 (+ 760us) client_negotiation.cc:240] Negotiation successful
0924 03:10:04.225274 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":870,"thread_start_us":55,"threads_started":1}
I20260924 03:10:04.225522 12105 raft_consensus.cc:504] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:04.225646 12105 leader_election.cc:288] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [CANDIDATE]: Term 1 election: Requested vote from peers 52d1aabbe4814a8bba1f1ba487d1282d (127.0.98.1:44247), 7792c1539cd04d83b1d5272db5bf7fc2 (127.0.98.3:46611)
I20260924 03:10:04.225658 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4de175d3ab4490b9b8b5a445688355" candidate_uuid: "68ac7ff126344a7f96c24ddc770d2221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" is_pre_election: true
I20260924 03:10:04.225801 12036 raft_consensus.cc:2454] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ac7ff126344a7f96c24ddc770d2221 in term 0.
I20260924 03:10:04.225803 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4de175d3ab4490b9b8b5a445688355" candidate_uuid: "68ac7ff126344a7f96c24ddc770d2221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792c1539cd04d83b1d5272db5bf7fc2"
I20260924 03:10:04.225920 12035 raft_consensus.cc:3037] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:04.226028 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc4de175d3ab4490b9b8b5a445688355" candidate_uuid: "68ac7ff126344a7f96c24ddc770d2221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d1aabbe4814a8bba1f1ba487d1282d"
I20260924 03:10:04.226092 11763 raft_consensus.cc:3037] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:04.227355 11763 raft_consensus.cc:2454] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ac7ff126344a7f96c24ddc770d2221 in term 1.
I20260924 03:10:04.227434 12035 raft_consensus.cc:2454] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ac7ff126344a7f96c24ddc770d2221 in term 1.
I20260924 03:10:04.227613 11840 leader_election.cc:302] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ac7ff126344a7f96c24ddc770d2221, 7792c1539cd04d83b1d5272db5bf7fc2; no voters:
I20260924 03:10:04.227684 12105 raft_consensus.cc:2781] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:04.227847 12105 raft_consensus.cc:686] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 1 LEADER]: Becoming Leader. State: Replica: 68ac7ff126344a7f96c24ddc770d2221, State: Running, Role: LEADER
I20260924 03:10:04.227960 12105 consensus_queue.cc:237] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } }
I20260924 03:10:04.229735 11601 catalog_manager.cc:5495] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ac7ff126344a7f96c24ddc770d2221 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "68ac7ff126344a7f96c24ddc770d2221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ac7ff126344a7f96c24ddc770d2221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 } health_report { overall_health: UNKNOWN } } }
I20260924 03:10:04.240221 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.237266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.98.2:35145)
0924 03:10:04.237309 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:04.237311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:04.237320 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:04.237387 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:04.237388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:04.237407 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:04.237472 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:04.237646 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.238339 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.239357 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:04.239631 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:04.239720 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:04.239885 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:04.239977 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:04.240094 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:04.240112 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 03:10:04.240131 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:10:04.241632 12035 raft_consensus.cc:1261] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 1 FOLLOWER]: Refusing update from remote peer 68ac7ff126344a7f96c24ddc770d2221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:04.241707 11763 raft_consensus.cc:1261] T cc4de175d3ab4490b9b8b5a445688355 P 52d1aabbe4814a8bba1f1ba487d1282d [term 1 FOLLOWER]: Refusing update from remote peer 68ac7ff126344a7f96c24ddc770d2221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:04.241991 12114 consensus_queue.cc:1035] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7792c1539cd04d83b1d5272db5bf7fc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:04.242081 12105 consensus_queue.cc:1035] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52d1aabbe4814a8bba1f1ba487d1282d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:04.244529 12117 mvcc.cc:205] Tried to move back new op lower bound from 7332738679771611136 to 7332738679718309888. Current Snapshot: MvccSnapshot[applied={T|T < 7332738679771611136}]
I20260924 03:10:04.245862 12119 mvcc.cc:205] Tried to move back new op lower bound from 7332738679771611136 to 7332738679718309888. Current Snapshot: MvccSnapshot[applied={T|T < 7332738679771611136}]
I20260924 03:10:04.246027 12118 mvcc.cc:205] Tried to move back new op lower bound from 7332738679771611136 to 7332738679718309888. Current Snapshot: MvccSnapshot[applied={T|T < 7332738679771611136}]
I20260924 03:10:04.248129 11600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:04.248211 11600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:04.248890 11600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c790ab87ce81477a9036f76c539c59bf: Sending DeleteTablet for 3 replicas of tablet cc4de175d3ab4490b9b8b5a445688355
I20260924 03:10:04.249222 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4de175d3ab4490b9b8b5a445688355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32800
I20260924 03:10:04.249223 12019 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4de175d3ab4490b9b8b5a445688355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43758
I20260924 03:10:04.249650 11872 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4de175d3ab4490b9b8b5a445688355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45330
I20260924 03:10:04.250087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 11682
I20260924 03:10:04.252822 12129 tablet_replica.cc:330] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: stopping tablet replica
I20260924 03:10:04.252964 12129 raft_consensus.cc:2229] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:10:04.253129 12129 raft_consensus.cc:2258] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:04.253616 12129 ts_tablet_manager.cc:1905] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:04.258131 12129 ts_tablet_manager.cc:1918] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:04.258271 12129 log.cc:1196] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/cc4de175d3ab4490b9b8b5a445688355
I20260924 03:10:04.258564 12129 ts_tablet_manager.cc:1939] T cc4de175d3ab4490b9b8b5a445688355 P 68ac7ff126344a7f96c24ddc770d2221: Deleting consensus metadata
I20260924 03:10:04.258958 11587 catalog_manager.cc:4841] TS 68ac7ff126344a7f96c24ddc770d2221 (127.0.98.2:35145): tablet cc4de175d3ab4490b9b8b5a445688355 (table test-table [id=69e7b8c4f96f44febcb3f76eeeca6607]) successfully deleted
I20260924 03:10:04.259087 12127 tablet_replica.cc:330] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: stopping tablet replica
I20260924 03:10:04.259184 12127 raft_consensus.cc:2229] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:04.259282 12127 raft_consensus.cc:2258] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:10:04.259338 11585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44247 (error 108)
I20260924 03:10:04.259690 12127 ts_tablet_manager.cc:1905] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:04.260133 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:04.260008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.0.1:32814)
0924 03:10:04.260053 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:04.260101 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 03:10:04.260129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 11819
W20260924 03:10:04.261159 11585 catalog_manager.cc:4584] TS 52d1aabbe4814a8bba1f1ba487d1282d (127.0.98.1:44247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc4de175d3ab4490b9b8b5a445688355: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44247: connect: Connection refused (error 111)
I20260924 03:10:04.262934 12127 ts_tablet_manager.cc:1918] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:04.263042 12127 log.cc:1196] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/cc4de175d3ab4490b9b8b5a445688355
I20260924 03:10:04.263329 12127 ts_tablet_manager.cc:1939] T cc4de175d3ab4490b9b8b5a445688355 P 7792c1539cd04d83b1d5272db5bf7fc2: Deleting consensus metadata
I20260924 03:10:04.263700 11588 catalog_manager.cc:4841] TS 7792c1539cd04d83b1d5272db5bf7fc2 (127.0.98.3:46611): tablet cc4de175d3ab4490b9b8b5a445688355 (table test-table [id=69e7b8c4f96f44febcb3f76eeeca6607]) successfully deleted
I20260924 03:10:04.265457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 11956
I20260924 03:10:04.271585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 11569
I20260924 03:10:04.275348 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:10:04.619102 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:10:04.619190 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:10:04.965294 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:10:04.965 UTC [11155] LOG: received smart shutdown request
2026-09-24 03:10:04.968 UTC [11155] LOG: background worker "logical replication launcher" (PID 11164) exited with exit code 1
2026-09-24 03:10:04.968 UTC [11159] LOG: shutting down
2026-09-24 03:10:05.125 UTC [11155] LOG: database system is shut down
2026-09-24T03:10:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41663 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:05 dist-test-slave-jtnz krb5kdc[12139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:05 dist-test-slave-jtnz krb5kdc[12139](info): set up 2 sockets
Sep 24 03:10:05 dist-test-slave-jtnz krb5kdc[12139](info): commencing operation
krb5kdc: starting...
W20260924 03:10:06.229983 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:10:06 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 03:10:06.368623 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskapYcIP/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/postgres -l logfile start
2026-09-24 03:10:08.519 UTC [12169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:10:08.520 UTC [12169] LOG: listening on IPv4 address "127.0.98.20", port 52343
2026-09-24 03:10:08.523 UTC [12169] LOG: listening on Unix socket "/tmp/.s.PGSQL.52343"
2026-09-24 03:10:08.556 UTC [12172] LOG: database system was shut down at 2026-09-24 03:10:07 UTC
2026-09-24 03:10:08.559 UTC [12169] LOG: database system is ready to accept connections
I20260924 03:10:09.453796 392 mini_postgres.cc:96] Postgres bound to 52343
2026-09-24 03:10:09.459 UTC [12180] FATAL: database "slave" does not exist
127.0.98.20:52343 - accepting connections
I20260924 03:10:09.459748 392 mini_ranger.cc:161] Starting Ranger...
I20260924 03:10:09.473424 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:10:09.876626 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:10:09.876706 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin
2026-09-24 03:10:10,119 [I] DB FLAVOR :POSTGRES
2026-09-24 03:10:10,120 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:10:10,120 [I] Checking connection..
2026-09-24 03:10:10,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:10:10,360 [I] Checking connection passed.
2026-09-24 03:10:10,361 [I] --------- Verifying version history table ---------
2026-09-24 03:10:10,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:10,616 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:10:10,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:10,839 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:10:10,840 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:10:10,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:10:11.054 UTC [12244] WARNING: there is no transaction in progress
2026-09-24 03:10:11,060 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:10:11,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:10:11,281 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:10:11,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:10:11,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:10:11,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:10:11,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jtnz.c.gcp-upstream.internal','N') ;"
2026-09-24 03:10:11,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:10:12,152 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:10:12,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:10:12,373 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:10:12,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:10:12,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:10:12,801 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:10:12,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:10:13.619 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.632 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.646 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.659 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.673 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.690 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.737 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.744 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.756 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.764 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.774 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.786 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.794 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.800 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.806 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:13.811 UTC [12352] WARNING: there is no transaction in progress
2026-09-24 03:10:14,348 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:10:14,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:10:14,576 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:10:14,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:10:14,814 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:10:14,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:10:15,024 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:10:15,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:10:15,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:10:15,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jtnz.c.gcp-upstream.internal';"
2026-09-24 03:10:15,718 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:10:15,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:10:15,940 [I] DB_PATCHES have already been applied
I20260924 03:10:15.945534 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskapYcIP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:10:15.945603 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 03:10:15.948302 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40463
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:10:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:10:16.989245 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.529s user 0.000s sys 0.008s
Sep 24, 2026 3:10:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:10:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:10:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:10:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:10:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:10:37.409292 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:10:37 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:10:37.489943 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:10:37.498479 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:10:38.216495 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:10:38.216571 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms
I20260924 03:10:38.216591 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52343
2026-09-24 03:10:38,367 [I] DB FLAVOR :POSTGRES
2026-09-24 03:10:38,367 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:10:38,368 [I] Checking connection
2026-09-24 03:10:38,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:10:38,582 [I] connection success
2026-09-24 03:10:38,583 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:10:38,583 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:10:38,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:10:38,808 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:10:38,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:10:38,808 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:10:38,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:10:39,031 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:10:39.424420 392 mini_ranger_kms.cc:310] Created kms service
I20260924 03:10:39.587510 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:10:39.631583 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:10:39 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:39.994652 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:10:39.994760 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskapYcIP/build/release/bin/postgresql.jar:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskapYcIP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:10:39.994776 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 03:10:39.997871 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12532
I20260924 03:10:39.997949 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39989
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 03:10:42 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:42.118412 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.628s user 0.008s sys 0.127s
I20260924 03:10:42.118516 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:10:42.118572 392 mini_ranger_kms.cc:401] 127.0.98.20:39989/kms/v1/keys
Sep 24 03:10:42 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:10:42 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:10:44 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:10:44.868867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42499
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:44.940135 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:44.940299 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:44.940317 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:44.941427 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:44.941468 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:44.941483 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:44.941519 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:44.941545 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:44.943070 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42499
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:44.943284 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:44.943478 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:44.947904 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:44.947999 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:44.948032 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:44.948042 12583 server_base.cc:1034] running on GCE node
I20260924 03:10:44.948345 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:44.948542 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:44.949697 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790219444949689 us; error 24 us; skew 500 ppm
Sep 24 03:10:44 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:44.958361 12583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:44.958758 12583 webserver.cc:460] Webserver started at http://127.0.98.62:35221/ using document root <none> and password file <none>
I20260924 03:10:44.958923 12583 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:44.958954 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:44.959030 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:44 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:45.003392 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "9deb51455d3243449fd6315ac93cba3a"
format_stamp: "Formatted at 2026-09-24 03:10:44 on dist-test-slave-jtnz"
server_key: "1fa6da887b5a98173d2d55ab7b1a2f17"
server_key_iv: "a3902a457277cb1dd42329559b2ca0e0"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.003799 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "9deb51455d3243449fd6315ac93cba3a"
format_stamp: "Formatted at 2026-09-24 03:10:44 on dist-test-slave-jtnz"
server_key: "1fa6da887b5a98173d2d55ab7b1a2f17"
server_key_iv: "a3902a457277cb1dd42329559b2ca0e0"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.005098 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:10:45.018428 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:45.018595 12583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:45.018740 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9deb51455d3243449fd6315ac93cba3a"
format_stamp: "Formatted at 2026-09-24 03:10:44 on dist-test-slave-jtnz"
server_key: "1fa6da887b5a98173d2d55ab7b1a2f17"
server_key_iv: "a3902a457277cb1dd42329559b2ca0e0"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.018913 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:45.028261 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:45.030148 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:45.030316 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:45.034730 12583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42499
I20260924 03:10:45.034806 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42499 every 8 connection(s)
I20260924 03:10:45.035115 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:45.035708 12651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:45.037746 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Bootstrap starting.
I20260924 03:10:45.038331 12651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:45.038599 12651 log.cc:824] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:45.039211 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: No bootstrap required, opened a new log
I20260924 03:10:45.041121 12651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:45.041252 12651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:45.041273 12651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9deb51455d3243449fd6315ac93cba3a, State: Initialized, Role: FOLLOWER
I20260924 03:10:45.041433 12651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:45.041507 12651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:45.041527 12651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:45.041603 12651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:45.042081 12651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:45.042157 12651 leader_election.cc:302] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9deb51455d3243449fd6315ac93cba3a; no voters:
I20260924 03:10:45.042384 12651 leader_election.cc:288] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:45.042415 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:45.042542 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 LEADER]: Becoming Leader. State: Replica: 9deb51455d3243449fd6315ac93cba3a, State: Running, Role: LEADER
I20260924 03:10:45.042623 12651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:45.042687 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:45.043052 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9deb51455d3243449fd6315ac93cba3a. Latest consensus state: current_term: 1 leader_uuid: "9deb51455d3243449fd6315ac93cba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260924 03:10:45.043318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 12583
I20260924 03:10:45.043412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:10:45.043571 392 external_mini_cluster.cc:1428] Setting key 358cf0a25170b23d17077f815130053d
I20260924 03:10:45.043201 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:45.044092 12651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:10:45.043135 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9deb51455d3243449fd6315ac93cba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260924 03:10:45.044270 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:45.046352 12668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:45.047333 12668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:45.058502 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:45.044040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.0.98.62:42499)
0924 03:10:45.047552 (+ 3512us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:45.047561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:45.047596 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:10:45.047722 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:45.047729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:45.047781 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:45.047932 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:45.048768 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.050258 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.050972 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.051646 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.051773 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:45.054412 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:45.054416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:45.054417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:45.054428 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:45.056517 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:45.056927 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:45.056929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:45.056930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:45.057021 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:45.057208 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:45.057209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:45.057209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:45.057504 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:45.057649 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:45.058142 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 03:10:45.058273 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3443,"thread_start_us":65,"threads_started":1}
I20260924 03:10:45.062095 12668 catalog_manager.cc:1347] Generated new cluster ID: 6e9d221066f0470cbc681e00b5235068
I20260924 03:10:45.062135 12668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:45.090128 12668 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:45.090903 12668 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:45.095831 12668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Generated new TSK 0
I20260924 03:10:45.096038 12668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:10:45.669157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:45.782686 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:45.782914 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:45.782943 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:45.784447 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:45.784503 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:45.784536 12696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:45.786540 12696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:45.786765 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:45.787024 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:45.791846 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:45.791863 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:45.792101 12696 server_base.cc:1034] running on GCE node
W20260924 03:10:45.792254 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:45.792461 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:45.792685 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:45.793824 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790219445793811 us; error 33 us; skew 500 ppm
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:45.796324 12696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:45.796715 12696 webserver.cc:460] Webserver started at http://127.0.98.1:44389/ using document root <none> and password file <none>
I20260924 03:10:45.796973 12696 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:45.797044 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:45.797168 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:45.809612 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b515ed74c804cf287191149bd79ebcb"
format_stamp: "Formatted at 2026-09-24 03:10:45 on dist-test-slave-jtnz"
server_key: "fad6ce8d8f40294b003b9d01fa568e6e"
server_key_iv: "f53404118c79438ab904769f912a1c0f"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.810065 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b515ed74c804cf287191149bd79ebcb"
format_stamp: "Formatted at 2026-09-24 03:10:45 on dist-test-slave-jtnz"
server_key: "fad6ce8d8f40294b003b9d01fa568e6e"
server_key_iv: "f53404118c79438ab904769f912a1c0f"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.812134 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260924 03:10:45.825834 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:45.826001 12696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:45.826077 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "9b515ed74c804cf287191149bd79ebcb"
format_stamp: "Formatted at 2026-09-24 03:10:45 on dist-test-slave-jtnz"
server_key: "fad6ce8d8f40294b003b9d01fa568e6e"
server_key_iv: "f53404118c79438ab904769f912a1c0f"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:45.826159 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:45.849709 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:45.851986 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:45.852133 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:45.852326 12696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:45.852645 12696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:45.852686 12696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:45.852715 12696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:45.852763 12696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:45.859580 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39711
I20260924 03:10:45.859984 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:45.860522 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39711 every 8 connection(s)
I20260924 03:10:45.866340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 12696
I20260924 03:10:45.866441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:45.866621 392 external_mini_cluster.cc:1428] Setting key d0fce4a7a56a03612a11b72bd07ca444
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:45.870774 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:45.861120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.1:58093)
0924 03:10:45.861289 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:45.861331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:45.861963 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:45.862262 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:45.862266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:45.862490 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:45.862736 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:45.862741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.863707 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.863708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:45.864806 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:45.864809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.865299 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.865300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:45.865435 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:45.865923 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:45.865932 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:45.867448 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:45.869183 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:45.869189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:45.869191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:45.869475 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:45.869684 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:45.869685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:45.869685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:45.869809 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:45.870093 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:45.870098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:45.870283 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 03:10:45.870379 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:10:45.870530 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260924 03:10:45.871467 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:45.861419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58093 (local address 127.0.98.62:42499)
0924 03:10:45.861548 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:45.861551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:45.861561 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:10:45.862079 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:45.862081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:45.862095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:45.862222 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:45.862838 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.863607 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.864898 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:45.865194 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:45.865268 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:45.867541 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:45.867543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:45.867543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:45.867551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:45.869088 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:45.869555 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:45.869556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:45.869556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:45.869617 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:45.869877 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:45.869878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:45.869878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:45.870019 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:45.870102 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:45.871241 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0924 03:10:45.871326 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260924 03:10:45.872236 12827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:45.872335 12827 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:45.872522 12827 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:45.873078 12615 ts_manager.cc:194] Registered new tserver with Master: 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711)
I20260924 03:10:45.873937 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58093
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:10:45.892937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:45.988428 12833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:45.988726 12833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:45.988761 12833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:45.989796 12833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:45.989853 12833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:45.989895 12833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:45.991336 12833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:45.991679 12833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:45.991916 12833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:45.996222 12843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:45.996222 12841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:45.996565 12840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:45.997421 12833 server_base.cc:1034] running on GCE node
I20260924 03:10:45.997607 12833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:45.997821 12833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:45.998996 12833 hybrid_clock.cc:648] HybridClock initialized: now 1790219445998965 us; error 22 us; skew 500 ppm
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.001976 12833 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:46.002485 12833 webserver.cc:460] Webserver started at http://127.0.98.2:46037/ using document root <none> and password file <none>
I20260924 03:10:46.002723 12833 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:46.002782 12833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:46.002908 12833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:45 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.019623 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c0ae24c4b6546ef98e521624297d463"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "9b6f629e23a9824951df8017d1fe4436"
server_key_iv: "23ef1aa09b7ab64a97f394b2dbff3eec"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.020031 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c0ae24c4b6546ef98e521624297d463"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "9b6f629e23a9824951df8017d1fe4436"
server_key_iv: "23ef1aa09b7ab64a97f394b2dbff3eec"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.021816 12833 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:46.045388 12850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.045629 12833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 03:10:46.045714 12833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "8c0ae24c4b6546ef98e521624297d463"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "9b6f629e23a9824951df8017d1fe4436"
server_key_iv: "23ef1aa09b7ab64a97f394b2dbff3eec"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.045785 12833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:46.082635 12833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:46.084798 12833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:46.085000 12833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:46.085227 12833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:46.085599 12833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:46.085692 12833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.085737 12833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:46.085754 12833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.092929 12833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39057
I20260924 03:10:46.092979 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39057 every 8 connection(s)
I20260924 03:10:46.093344 12833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:46.100905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 12833
I20260924 03:10:46.101004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:46.101225 392 external_mini_cluster.cc:1428] Setting key b14548b40983a8637bf5aa3dfbd46e1c
I20260924 03:10:46.102348 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.094318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51711 (local address 127.0.98.62:42499)
0924 03:10:46.094355 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:46.094357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:46.094367 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:10:46.094853 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:46.094855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:46.094870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:46.094927 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:46.095421 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.095980 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.096861 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.097076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.097160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.099098 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:46.099099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:46.099100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:46.099107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:46.100627 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.101176 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.101177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.101177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.101233 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.101555 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.101556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.101556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.101681 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:46.101720 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:46.102104 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 03:10:46.102214 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:10:46.102877 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.094037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.2:51711)
0924 03:10:46.094193 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:46.094229 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:46.094746 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:46.094994 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:46.094997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:46.095164 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:46.095333 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.095336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.096058 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.096059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.096775 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.096778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.097176 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.097177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.097283 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.097761 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:46.097769 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:46.098995 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:46.100739 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.100763 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.100765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.101056 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.101313 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.101314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.101314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.101473 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.101767 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:46.101772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:46.101946 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 03:10:46.102429 (+ 483us) client_negotiation.cc:240] Negotiation successful
0924 03:10:46.102619 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":86,"threads_started":1}
I20260924 03:10:46.104305 12965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:46.104426 12965 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.104599 12965 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:46.104993 12615 ts_manager.cc:194] Registered new tserver with Master: 8c0ae24c4b6546ef98e521624297d463 (127.0.98.2:39057)
I20260924 03:10:46.105352 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51711
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:10:46.126839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:46.231750 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:46.232059 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:46.232136 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:46.233613 12970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:46.233680 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:46.233726 12970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:46.235533 12970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:46.235849 12970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:46.236096 12970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:46.241619 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.241662 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.241811 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:46.242605 12970 server_base.cc:1034] running on GCE node
I20260924 03:10:46.242801 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:46.243062 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:46.244227 12970 hybrid_clock.cc:648] HybridClock initialized: now 1790219446244191 us; error 47 us; skew 500 ppm
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.246829 12970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:46.247197 12970 webserver.cc:460] Webserver started at http://127.0.98.3:43461/ using document root <none> and password file <none>
I20260924 03:10:46.247409 12970 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:46.247483 12970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:46.247640 12970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:46.259285 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e894773d44b4ab5bd762032231c2a99"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "538dc4371c7257691c16290c98ebdeb8"
server_key_iv: "415b4aa2aeae7219bd8ef68d11ea7dc8"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.259657 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e894773d44b4ab5bd762032231c2a99"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "538dc4371c7257691c16290c98ebdeb8"
server_key_iv: "415b4aa2aeae7219bd8ef68d11ea7dc8"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.261471 12970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:10:46.286459 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.286670 12970 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.001s
I20260924 03:10:46.286755 12970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "0e894773d44b4ab5bd762032231c2a99"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "538dc4371c7257691c16290c98ebdeb8"
server_key_iv: "415b4aa2aeae7219bd8ef68d11ea7dc8"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.286839 12970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:46.305204 12970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:46.307238 12970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:46.307408 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:46.307653 12970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:46.307999 12970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:46.308089 12970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.308157 12970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:46.308187 12970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.314508 12970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45753
I20260924 03:10:46.315008 12970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:46.316107 13098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45753 every 8 connection(s)
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:46.324499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 12970
I20260924 03:10:46.324587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:46.324765 392 external_mini_cluster.cc:1428] Setting key 79a7ee1d36587d43363c0326b2c1f492
I20260924 03:10:46.325336 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.316825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50447 (local address 127.0.98.62:42499)
0924 03:10:46.316866 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:46.316868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:46.316877 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:46.317857 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:46.317858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:46.317871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:46.317923 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:46.318492 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.318949 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.320008 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.320213 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.320307 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.322282 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:46.322283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:46.322284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:46.322291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:46.323803 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.324243 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.324244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.324244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.324298 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.324590 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.324591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.324591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.324714 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:46.324769 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:46.325156 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 03:10:46.325243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:10:46.326040 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.316597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.3:50447)
0924 03:10:46.316747 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:46.316783 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:46.317745 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:46.317985 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:46.317989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:46.318230 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:46.318411 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.318415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.319054 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.319055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.319918 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.319920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.320327 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.320329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.320436 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.320885 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:46.320892 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:46.322169 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:46.323890 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.323895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.323897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.324158 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.324369 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.324370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.324370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.324503 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.324814 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:46.324820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:46.324996 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 03:10:46.325659 (+ 663us) client_negotiation.cc:240] Negotiation successful
0924 03:10:46.325808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260924 03:10:46.326573 13099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:46.326668 13099 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.326877 13099 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:46.327286 12615 ts_manager.cc:194] Registered new tserver with Master: 0e894773d44b4ab5bd762032231c2a99 (127.0.98.3:45753)
I20260924 03:10:46.327755 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50447
I20260924 03:10:46.336146 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:10:46.336787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 12696
I20260924 03:10:46.342538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 12833
I20260924 03:10:46.347864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 12970
I20260924 03:10:46.353070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 12583
I20260924 03:10:46.357968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--webserver_interface=127.0.98.62
--webserver_port=35221
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42499
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:46.430821 13102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:46.431003 13102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:46.431035 13102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:46.432214 13102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:46.432271 13102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:46.432296 13102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:46.432325 13102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:46.432346 13102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:46.433728 13102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42499
--ranger_config_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:46.433939 13102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:46.434122 13102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:46.438333 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.438390 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:46.438494 13102 server_base.cc:1034] running on GCE node
W20260924 03:10:46.438627 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:46.442050 13102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:46.442422 13102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:46.452437 13102 hybrid_clock.cc:648] HybridClock initialized: now 1790219446452415 us; error 34 us; skew 500 ppm
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.456223 13102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:46.456727 13102 webserver.cc:460] Webserver started at http://127.0.98.62:35221/ using document root <none> and password file <none>
I20260924 03:10:46.456992 13102 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:46.457063 13102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:46.501909 13102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:46.502691 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.502849 13102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.502933 13102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9deb51455d3243449fd6315ac93cba3a"
format_stamp: "Formatted at 2026-09-24 03:10:44 on dist-test-slave-jtnz"
server_key: "1fa6da887b5a98173d2d55ab7b1a2f17"
server_key_iv: "a3902a457277cb1dd42329559b2ca0e0"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.503268 13102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:46.536325 13102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:46.538874 13102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:46.539055 13102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:46.543779 13102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42499
I20260924 03:10:46.543807 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42499 every 8 connection(s)
I20260924 03:10:46.544164 13102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:46.545315 13171 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:10:46.545873 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Bootstrap starting.
I20260924 03:10:46.552910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13102
I20260924 03:10:46.553154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39711
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42499
--webserver_port=44389
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:46.553570 13171 log.cc:824] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:46.555625 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:46.555871 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Bootstrap complete.
I20260924 03:10:46.557937 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:46.558161 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9deb51455d3243449fd6315ac93cba3a, State: Initialized, Role: FOLLOWER
I20260924 03:10:46.558364 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:46.558434 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:46.558480 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:46.558527 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:10:46.559442 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:46.559563 13171 leader_election.cc:302] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9deb51455d3243449fd6315ac93cba3a; no voters:
I20260924 03:10:46.559702 13171 leader_election.cc:288] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:10:46.559736 13175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:10:46.559831 13175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [term 2 LEADER]: Becoming Leader. State: Replica: 9deb51455d3243449fd6315ac93cba3a, State: Running, Role: LEADER
I20260924 03:10:46.559940 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:46.559980 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260924 03:10:46.560431 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9deb51455d3243449fd6315ac93cba3a. Latest consensus state: current_term: 2 leader_uuid: "9deb51455d3243449fd6315ac93cba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260924 03:10:46.560519 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:46.560709 13176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9deb51455d3243449fd6315ac93cba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9deb51455d3243449fd6315ac93cba3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260924 03:10:46.560801 13176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:46.561512 13171 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:10:46.563606 13181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:46.564634 13181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:46.564836 13181 catalog_manager.cc:1259] Loaded cluster ID: 6e9d221066f0470cbc681e00b5235068
I20260924 03:10:46.564873 13181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:46.566944 13181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:46.567212 13181 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Loaded TSK: 0
I20260924 03:10:46.567504 13181 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:10:46.658454 13174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:46.658814 13174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:46.658890 13174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:46.660506 13174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:46.660588 13174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:46.660665 13174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:46.662789 13174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44389
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:46.663089 13174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:46.663300 13174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:46.668817 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.668776 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.668780 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:46.669193 13174 server_base.cc:1034] running on GCE node
I20260924 03:10:46.669495 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:46.669757 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:46.670920 13174 hybrid_clock.cc:648] HybridClock initialized: now 1790219446670903 us; error 33 us; skew 500 ppm
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.674031 13174 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:46.674563 13174 webserver.cc:460] Webserver started at http://127.0.98.1:44389/ using document root <none> and password file <none>
I20260924 03:10:46.674805 13174 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:46.674886 13174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:46.696647 13174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:46.697345 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.697518 13174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:46.697599 13174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "9b515ed74c804cf287191149bd79ebcb"
format_stamp: "Formatted at 2026-09-24 03:10:45 on dist-test-slave-jtnz"
server_key: "fad6ce8d8f40294b003b9d01fa568e6e"
server_key_iv: "f53404118c79438ab904769f912a1c0f"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.697968 13174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:46.718799 13174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:46.720599 13174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:46.720722 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:46.720932 13174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:46.721248 13174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:46.721305 13174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.721344 13174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:46.721383 13174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.728269 13174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39711
I20260924 03:10:46.728425 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39711 every 8 connection(s)
I20260924 03:10:46.728745 13174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:46.730373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13174
I20260924 03:10:46.730629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39057
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42499
--webserver_port=46037
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:46.745900 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.729667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.1:50077)
0924 03:10:46.731673 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:46.731716 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:46.732605 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:46.733252 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:46.733256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:46.733499 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:46.733747 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.733752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.735261 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.735262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.736446 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.736449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.737278 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.737279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.737406 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.740105 (+ 2699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:46.740114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:46.741707 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:46.744070 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.744076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.744078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.744371 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.744655 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.744656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.744656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.744771 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.745268 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:46.745274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:46.745425 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 03:10:46.745534 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:10:46.745681 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1967,"thread_start_us":69,"threads_started":1}
I20260924 03:10:46.746834 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.731375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50077 (local address 127.0.98.62:42499)
0924 03:10:46.731606 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:46.731616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:46.732070 (+ 454us) server_negotiation.cc:407] Connection header received
0924 03:10:46.732908 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:46.732915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:46.732977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:46.733105 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:46.733867 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.735144 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.736539 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.737086 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.737187 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.741849 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:46.741853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:46.741854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:46.741867 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:46.743971 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.744480 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.744483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.744484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.744584 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.744838 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.744839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.744840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.745157 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:46.745239 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:46.746519 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0924 03:10:46.746647 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260924 03:10:46.747656 13333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:46.747768 13333 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.747941 13333 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:46.748733 13333 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:10:46.748814 13333 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.748881 13333 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:46.749209 13333 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.749336 13333 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:46.749610 13333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:10:46.844542 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:46.844767 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:46.844796 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:46.846297 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:46.846376 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:46.846429 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:46.848320 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46037
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:46.848531 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:46.848771 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:46.854339 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.854355 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:46.854362 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:46.855201 13337 server_base.cc:1034] running on GCE node
I20260924 03:10:46.855399 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:46.855620 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:46.856810 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790219446856795 us; error 28 us; skew 500 ppm
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:46.859735 13337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:46.860148 13337 webserver.cc:460] Webserver started at http://127.0.98.2:46037/ using document root <none> and password file <none>
I20260924 03:10:46.860412 13337 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:46.860497 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:46.880798 13337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:46.882329 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.882552 13337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:46.882639 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "8c0ae24c4b6546ef98e521624297d463"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "9b6f629e23a9824951df8017d1fe4436"
server_key_iv: "23ef1aa09b7ab64a97f394b2dbff3eec"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:46.883031 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:46.903821 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:46.906031 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:46.906230 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:46.906478 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:46.906824 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:46.906914 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.906967 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:46.907011 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:46.914448 13337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39057
I20260924 03:10:46.914923 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:46.922251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13337
I20260924 03:10:46.922534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45753
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42499
--webserver_port=43461
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:10:46 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:46.926291 13466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39057 every 8 connection(s)
I20260924 03:10:46.929174 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.916761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60809 (local address 127.0.98.62:42499)
0924 03:10:46.916805 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:46.916807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:46.916816 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:46.918009 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:46.918011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:46.918026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:46.918087 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:46.918732 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.919476 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.921088 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.921435 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.921514 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.926128 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:46.926129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:46.926130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:46.926136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:46.927666 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.928130 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.928131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.928132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.928189 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:46.928481 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:46.928482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:46.928482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:46.928607 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:46.928653 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:46.929013 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 03:10:46.929091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:10:46.929607 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:46.916479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.2:60809)
0924 03:10:46.916655 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:46.916695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:46.917879 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:46.918166 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:46.918170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:46.918396 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:46.918631 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.918638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.919581 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.919582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.920990 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:46.920993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:46.921594 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:46.921595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:46.921736 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:46.924087 (+ 2351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:46.924096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:46.926026 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:46.927769 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.927775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.927777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.928037 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.928264 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:46.928264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:46.928264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:46.928401 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:46.928702 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:46.928709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:46.928855 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 03:10:46.929262 (+ 407us) client_negotiation.cc:240] Negotiation successful
0924 03:10:46.929412 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260924 03:10:46.930114 13467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:46.930230 13467 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.930406 13467 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:46.931057 13467 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:10:46.931133 13467 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.931187 13467 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:46.931546 13467 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:46.931631 13467 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:46.931958 13467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:10:47.037973 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:47.038273 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:47.038307 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:47.039911 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:47.040004 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:47.040086 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:47.042466 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43461
--tserver_master_addrs=127.0.98.62:42499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:47.042697 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:47.043004 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:47.049382 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:47.049690 13470 server_base.cc:1034] running on GCE node
W20260924 03:10:47.050593 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:47.051030 13476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:47.051342 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:47.051600 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:47.052765 13470 hybrid_clock.cc:648] HybridClock initialized: now 1790219447052732 us; error 46 us; skew 500 ppm
Sep 24 03:10:47 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:47.055496 13470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:47.056023 13470 webserver.cc:460] Webserver started at http://127.0.98.3:43461/ using document root <none> and password file <none>
I20260924 03:10:47.056236 13470 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:47.056313 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:10:47 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:47.077783 13470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:10:47.078500 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:47.078680 13470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:47.078761 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "0e894773d44b4ab5bd762032231c2a99"
format_stamp: "Formatted at 2026-09-24 03:10:46 on dist-test-slave-jtnz"
server_key: "538dc4371c7257691c16290c98ebdeb8"
server_key_iv: "415b4aa2aeae7219bd8ef68d11ea7dc8"
server_key_version: "kuduclusterkey@0"
I20260924 03:10:47.079173 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:47.098987 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:47.101145 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:47.101364 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:47.101640 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:47.102016 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:47.102087 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:47.102134 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:47.102169 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:47.108532 13470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45753
I20260924 03:10:47.109021 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:47.114027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13470
I20260924 03:10:47.118343 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45753 every 8 connection(s)
I20260924 03:10:47.130596 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:47.114438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.0.98.62:42499)
0924 03:10:47.114484 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:47.114486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:47.122451 (+ 7965us) server_negotiation.cc:407] Connection header received
0924 03:10:47.122501 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:47.122503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:47.122517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:47.122594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:47.123624 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.124375 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.125471 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.125732 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.125805 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:47.127233 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:47.127234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:47.127235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:47.127241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:47.128913 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:47.129302 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:47.129303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:47.129304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:47.129368 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:47.130006 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:47.130007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:47.130007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:47.130136 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:47.130324 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:47.130412 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 03:10:47.130504 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 03:10:47 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:47.138067 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:47.123194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42499 (local address 127.0.98.3:33245)
0924 03:10:47.124508 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:47.124556 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:47.125153 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:47.125917 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:47.125921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:47.126148 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:47.126446 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:47.126462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.129054 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.129056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:47.131852 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:47.131855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.132372 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.132373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:47.132534 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:47.132993 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:47.133000 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:47.134596 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:47.136501 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:47.136506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:47.136508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:47.136800 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:47.136996 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:47.136997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:47.136997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:47.137113 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:47.137404 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:47.137410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:47.137585 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 03:10:47.137692 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 03:10:47.137844 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":90,"threads_started":1}
I20260924 03:10:47.138592 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:47.123487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33245 (local address 127.0.98.62:42499)
0924 03:10:47.123685 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:47.123687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:47.125289 (+ 1602us) server_negotiation.cc:407] Connection header received
0924 03:10:47.125312 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:47.125313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:47.125326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:47.125378 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:47.126556 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.128249 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.131946 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:47.132235 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:47.132297 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:47.134698 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:47.134699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:47.134699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:47.134705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:47.136405 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:47.136884 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:47.136885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:47.136885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:47.136937 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:47.137177 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:47.137178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:47.137178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:47.137318 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:47.137377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:47.138396 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0924 03:10:47.138503 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":57,"threads_started":1}
I20260924 03:10:47.139195 13601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42499
I20260924 03:10:47.139317 13601 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:47.139547 13601 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:47.140120 13601 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:10:47.140237 13601 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:47.140353 13601 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:47.140681 13601 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:47.140785 13601 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
W20260924 03:10:47.141063 13601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 03:10:47 dist-test-slave-jtnz krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:47.749796 13333 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:47.749971 13333 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:47.750330 13135 ts_manager.cc:194] Registered new tserver with Master: 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711)
I20260924 03:10:47.751312 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50077
Sep 24 03:10:47 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:47.932139 13467 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:47.932269 13467 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:47.932610 13135 ts_manager.cc:194] Registered new tserver with Master: 8c0ae24c4b6546ef98e521624297d463 (127.0.98.2:39057)
I20260924 03:10:47.933002 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60809
Sep 24 03:10:48 dist-test-slave-jtnz krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:10:48.141258 13601 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:48.141444 13601 heartbeater.cc:511] Master 127.0.98.62:42499 requested a full tablet report, sending...
I20260924 03:10:48.141783 13135 ts_manager.cc:194] Registered new tserver with Master: 0e894773d44b4ab5bd762032231c2a99 (127.0.98.3:45753)
I20260924 03:10:48.142405 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33245
I20260924 03:10:48.145218 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:10:48.154626 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.146095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.98.62:42499)
0924 03:10:48.146790 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.146793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.146800 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:48.146821 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.146822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.146833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.146894 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:48.147591 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.148300 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.149223 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.149494 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.149553 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.151033 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:48.151035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:48.151035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:48.151042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:48.152744 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:48.153642 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:48.153643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:48.153643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:48.153703 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:48.153902 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:48.153903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:48.153903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:48.154016 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:48.154097 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.154466 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.154540 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":61,"threads_started":1}
I20260924 03:10:48.161123 13135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:10:48.163355 13135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:10:48.171644 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39711 (local address 127.0.0.1:57610)
0924 03:10:48.167179 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.167188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.167216 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.167573 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.167573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.167575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.167617 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.167618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.168944 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.168945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.170033 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.170035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.171372 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.171373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.171459 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.171474 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.171522 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.171567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260924 03:10:48.171800 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.0.98.3:45753)
0924 03:10:48.166692 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.166696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.167331 (+ 635us) server_negotiation.cc:407] Connection header received
0924 03:10:48.167373 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.167375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.167389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.167464 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.167702 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.168630 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.170104 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.170844 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.170909 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.170985 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.171691 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.171729 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":47,"threads_started":1}
I20260924 03:10:48.172331 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.98.1:39711)
0924 03:10:48.167388 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.167392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.167401 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:48.167429 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.167431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.167443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.167507 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.167690 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.168625 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.170134 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.171266 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.172099 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.172195 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.172220 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.172259 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":61,"threads_started":1}
I20260924 03:10:48.172601 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39057 (local address 127.0.0.1:50168)
0924 03:10:48.166743 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.166778 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.167070 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.168951 (+ 1881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.168952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.168954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.168985 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.168987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.170218 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.170219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.171352 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.171354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.172365 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.172366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.172441 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.172445 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.172489 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.172525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":107,"threads_started":1}
I20260924 03:10:48.172605 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45753 (local address 127.0.0.1:58002)
0924 03:10:48.167266 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.167271 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.167305 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.167551 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.167553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.167555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.167614 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.167616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.168721 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.168722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.170012 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.170015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.171504 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.171505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.171561 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.171565 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.171612 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.171647 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":58,"threads_started":1}
I20260924 03:10:48.173158 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.166326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.0.98.2:39057)
0924 03:10:48.166646 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.166651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.166871 (+ 220us) server_negotiation.cc:407] Connection header received
0924 03:10:48.168844 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.168847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.168862 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.169068 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.169136 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.170109 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.171462 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.172270 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.172944 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.173025 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.173050 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.173085 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":56,"threads_started":1}
I20260924 03:10:48.173226 13267 tablet_service.cc:1467] Processing CreateTablet for tablet 3428d04e614e495e8853baf135a832d1 (DEFAULT_TABLE table=test-table [id=0ce014cc5678461ba5c37cc621ff9bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:48.173601 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428d04e614e495e8853baf135a832d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:48.173866 13401 tablet_service.cc:1467] Processing CreateTablet for tablet 3428d04e614e495e8853baf135a832d1 (DEFAULT_TABLE table=test-table [id=0ce014cc5678461ba5c37cc621ff9bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:48.174145 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428d04e614e495e8853baf135a832d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:48.177380 13523 tablet_service.cc:1467] Processing CreateTablet for tablet 3428d04e614e495e8853baf135a832d1 (DEFAULT_TABLE table=test-table [id=0ce014cc5678461ba5c37cc621ff9bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:48.177663 13523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428d04e614e495e8853baf135a832d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:48.177816 13637 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: Bootstrap starting.
I20260924 03:10:48.178443 13637 tablet_bootstrap.cc:654] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:48.178730 13637 log.cc:824] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:48.179415 13637 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: No bootstrap required, opened a new log
I20260924 03:10:48.179482 13637 ts_tablet_manager.cc:1397] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:48.182036 13637 raft_consensus.cc:348] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.182567 13637 raft_consensus.cc:374] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:48.182653 13637 raft_consensus.cc:729] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0ae24c4b6546ef98e521624297d463, State: Initialized, Role: FOLLOWER
I20260924 03:10:48.182816 13637 consensus_queue.cc:260] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.183099 13637 ts_tablet_manager.cc:1428] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260924 03:10:48.183234 13467 heartbeater.cc:503] Master 127.0.98.62:42499 was elected leader, sending a full tablet report...
I20260924 03:10:48.184321 13639 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Bootstrap starting.
I20260924 03:10:48.184899 13639 tablet_bootstrap.cc:654] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:48.185175 13639 log.cc:824] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:48.186299 13636 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Bootstrap starting.
I20260924 03:10:48.188557 13636 tablet_bootstrap.cc:654] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:48.188865 13636 log.cc:824] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:48.194571 13639 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: No bootstrap required, opened a new log
I20260924 03:10:48.194676 13639 ts_tablet_manager.cc:1397] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 03:10:48.197540 13639 raft_consensus.cc:348] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.198681 13636 tablet_bootstrap.cc:492] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: No bootstrap required, opened a new log
I20260924 03:10:48.198788 13636 ts_tablet_manager.cc:1397] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 03:10:48.201375 13636 raft_consensus.cc:348] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.201589 13639 raft_consensus.cc:374] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:48.201676 13636 raft_consensus.cc:374] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:48.201756 13639 raft_consensus.cc:729] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e894773d44b4ab5bd762032231c2a99, State: Initialized, Role: FOLLOWER
I20260924 03:10:48.201994 13639 consensus_queue.cc:260] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.201830 13636 raft_consensus.cc:729] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b515ed74c804cf287191149bd79ebcb, State: Initialized, Role: FOLLOWER
I20260924 03:10:48.206266 13636 consensus_queue.cc:260] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.206617 13601 heartbeater.cc:503] Master 127.0.98.62:42499 was elected leader, sending a full tablet report...
I20260924 03:10:48.206728 13333 heartbeater.cc:503] Master 127.0.98.62:42499 was elected leader, sending a full tablet report...
I20260924 03:10:48.207365 13639 ts_tablet_manager.cc:1428] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260924 03:10:48.207496 13636 ts_tablet_manager.cc:1428] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
W20260924 03:10:48.232820 13334 tablet.cc:2369] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:10:48.291465 13643 raft_consensus.cc:482] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:48.291635 13643 raft_consensus.cc:504] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.292119 13643 leader_election.cc:288] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711), 8c0ae24c4b6546ef98e521624297d463 (127.0.98.2:39057)
I20260924 03:10:48.292598 13640 raft_consensus.cc:482] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:48.292706 13640 raft_consensus.cc:504] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.293054 13640 leader_election.cc:288] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e894773d44b4ab5bd762032231c2a99 (127.0.98.3:45753), 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711)
I20260924 03:10:48.300025 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.292433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55221 (local address 127.0.98.1:39711)
0924 03:10:48.292479 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.292482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.293615 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 03:10:48.293723 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.293724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.293736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.293765 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.294027 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.294764 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.296662 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.297599 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.297647 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.297658 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.299905 (+ 2247us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.299958 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:10:48.300422 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.292258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39711 (local address 127.0.98.3:55221)
0924 03:10:48.293548 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.293560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.293643 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.293837 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.293838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.293840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.293929 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.293930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.294863 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.294864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.296173 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.296176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.297718 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.297719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.297788 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.297791 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.300287 (+ 2496us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.300328 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"thread_start_us":80,"threads_started":1}
I20260924 03:10:48.300823 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "0e894773d44b4ab5bd762032231c2a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b515ed74c804cf287191149bd79ebcb" is_pre_election: true
I20260924 03:10:48.300956 13287 raft_consensus.cc:2454] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e894773d44b4ab5bd762032231c2a99 in term 0.
I20260924 03:10:48.301301 13488 leader_election.cc:302] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e894773d44b4ab5bd762032231c2a99, 9b515ed74c804cf287191149bd79ebcb; no voters:
I20260924 03:10:48.301431 13643 raft_consensus.cc:2781] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:48.301486 13643 raft_consensus.cc:482] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:48.301508 13643 raft_consensus.cc:3037] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:48.302600 13643 raft_consensus.cc:504] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.302737 13643 leader_election.cc:288] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [CANDIDATE]: Term 1 election: Requested vote from peers 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711), 8c0ae24c4b6546ef98e521624297d463 (127.0.98.2:39057)
I20260924 03:10:48.302842 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "0e894773d44b4ab5bd762032231c2a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b515ed74c804cf287191149bd79ebcb"
I20260924 03:10:48.302911 13287 raft_consensus.cc:3037] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:48.303122 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57525 (local address 127.0.98.2:39057)
0924 03:10:48.293551 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.293553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.296338 (+ 2785us) server_negotiation.cc:407] Connection header received
0924 03:10:48.296422 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.296423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.296433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.296462 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.297880 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.298560 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.300663 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.301147 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.301194 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.301205 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.303002 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.303064 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:10:48.303318 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39057 (local address 127.0.98.3:57525)
0924 03:10:48.296285 (+ 2948us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.296294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.296347 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.296520 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.296521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.296522 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.296585 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.296586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.298659 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.298660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.299817 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.299819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.301277 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.301278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.301358 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.301361 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.303211 (+ 1850us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.303252 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2931,"thread_start_us":70,"threads_started":1}
I20260924 03:10:48.303603 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "0e894773d44b4ab5bd762032231c2a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0ae24c4b6546ef98e521624297d463" is_pre_election: true
I20260924 03:10:48.303697 13404 raft_consensus.cc:2454] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e894773d44b4ab5bd762032231c2a99 in term 0.
I20260924 03:10:48.303787 13287 raft_consensus.cc:2454] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e894773d44b4ab5bd762032231c2a99 in term 1.
I20260924 03:10:48.303807 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "0e894773d44b4ab5bd762032231c2a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0ae24c4b6546ef98e521624297d463"
I20260924 03:10:48.303860 13421 raft_consensus.cc:3037] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:48.303938 13488 leader_election.cc:302] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e894773d44b4ab5bd762032231c2a99, 9b515ed74c804cf287191149bd79ebcb; no voters:
I20260924 03:10:48.304270 13643 raft_consensus.cc:2781] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:48.304409 13643 raft_consensus.cc:686] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 1 LEADER]: Becoming Leader. State: Replica: 0e894773d44b4ab5bd762032231c2a99, State: Running, Role: LEADER
I20260924 03:10:48.304503 13643 consensus_queue.cc:237] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } }
I20260924 03:10:48.304737 13421 raft_consensus.cc:2454] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e894773d44b4ab5bd762032231c2a99 in term 1.
I20260924 03:10:48.305367 13135 catalog_manager.cc:5495] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e894773d44b4ab5bd762032231c2a99 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e894773d44b4ab5bd762032231c2a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e894773d44b4ab5bd762032231c2a99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 } health_report { overall_health: UNKNOWN } } }
I20260924 03:10:48.307787 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36547 (local address 127.0.98.3:45753)
0924 03:10:48.293583 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.293585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.298770 (+ 5185us) server_negotiation.cc:407] Connection header received
0924 03:10:48.298856 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.298857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.298865 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.298890 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.299086 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.299732 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.306119 (+ 6387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.306658 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.306720 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.306728 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.307696 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.307732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:10:48.307987 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45753 (local address 127.0.98.2:36547)
0924 03:10:48.298714 (+ 5555us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.298723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.298793 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.298940 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.298941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.298942 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.299015 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.299016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.299813 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.299814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.306048 (+ 6234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.306050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.307542 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.307543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.307608 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.307611 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.307852 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.307898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5537,"mutex_wait_us":71,"thread_start_us":91,"threads_started":1}
I20260924 03:10:48.308118 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48205 (local address 127.0.98.1:39711)
0924 03:10:48.300100 (+ 6451us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.300102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.301614 (+ 1512us) server_negotiation.cc:407] Connection header received
0924 03:10:48.301701 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.301702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.301710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.301736 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:48.301906 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.302581 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.307104 (+ 4523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.307629 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.307682 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.307690 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.308005 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.308048 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6438,"thread_start_us":71,"threads_started":1}
I20260924 03:10:48.308331 13536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "8c0ae24c4b6546ef98e521624297d463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e894773d44b4ab5bd762032231c2a99" is_pre_election: true
I20260924 03:10:48.308411 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.293347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39711 (local address 127.0.98.2:48205)
0924 03:10:48.301554 (+ 8207us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.301565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:48.301637 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:48.301787 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:48.301787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:48.301788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:48.301838 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.301839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.302713 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.302714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.307001 (+ 4287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:48.307004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.307717 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.307718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:48.307780 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.307783 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:48.308281 (+ 498us) client_negotiation.cc:240] Negotiation successful
0924 03:10:48.308328 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8188,"thread_start_us":61,"threads_started":1}
I20260924 03:10:48.308600 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428d04e614e495e8853baf135a832d1" candidate_uuid: "8c0ae24c4b6546ef98e521624297d463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b515ed74c804cf287191149bd79ebcb" is_pre_election: true
I20260924 03:10:48.308677 13287 raft_consensus.cc:2379] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c0ae24c4b6546ef98e521624297d463 in current term 1: Already voted for candidate 0e894773d44b4ab5bd762032231c2a99 in this term.
I20260924 03:10:48.308872 13354 leader_election.cc:302] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c0ae24c4b6546ef98e521624297d463; no voters: 0e894773d44b4ab5bd762032231c2a99, 9b515ed74c804cf287191149bd79ebcb
I20260924 03:10:48.309042 13640 raft_consensus.cc:2726] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 03:10:48.318691 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.315427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.0.98.3:45753)
0924 03:10:48.315477 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:48.315479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:48.315486 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:48.315516 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:48.315516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:48.315533 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:48.315598 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:48.315762 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.316454 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.317889 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:48.318089 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:48.318169 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:48.318353 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:48.318462 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:48.318568 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:48.318592 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:10:48.318609 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:10:48.320330 13421 raft_consensus.cc:1261] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463 [term 1 FOLLOWER]: Refusing update from remote peer 0e894773d44b4ab5bd762032231c2a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:48.320339 13287 raft_consensus.cc:1261] T 3428d04e614e495e8853baf135a832d1 P 9b515ed74c804cf287191149bd79ebcb [term 1 FOLLOWER]: Refusing update from remote peer 0e894773d44b4ab5bd762032231c2a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:48.320709 13643 consensus_queue.cc:1035] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0ae24c4b6546ef98e521624297d463" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:48.320814 13654 consensus_queue.cc:1035] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b515ed74c804cf287191149bd79ebcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:48.323586 13653 mvcc.cc:205] Tried to move back new op lower bound from 7332738860317839360 to 7332738860255870976. Current Snapshot: MvccSnapshot[applied={T|T < 7332738860317839360}]
I20260924 03:10:48.324644 13655 mvcc.cc:205] Tried to move back new op lower bound from 7332738860317839360 to 7332738860255870976. Current Snapshot: MvccSnapshot[applied={T|T < 7332738860317839360}]
I20260924 03:10:48.326486 13656 mvcc.cc:205] Tried to move back new op lower bound from 7332738860317839360 to 7332738860255870976. Current Snapshot: MvccSnapshot[applied={T|T < 7332738860317839360}]
I20260924 03:10:48.328578 13134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:48.328661 13134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:48.329389 13134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9deb51455d3243449fd6315ac93cba3a: Sending DeleteTablet for 3 replicas of tablet 3428d04e614e495e8853baf135a832d1
I20260924 03:10:48.329792 13267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428d04e614e495e8853baf135a832d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57610
I20260924 03:10:48.330169 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428d04e614e495e8853baf135a832d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50168
I20260924 03:10:48.330559 13523 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428d04e614e495e8853baf135a832d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260924 03:10:48.330868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13174
I20260924 03:10:48.334301 13665 tablet_replica.cc:330] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: stopping tablet replica
I20260924 03:10:48.334496 13665 raft_consensus.cc:2229] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:10:48.334681 13665 raft_consensus.cc:2258] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:48.335160 13665 ts_tablet_manager.cc:1905] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:48.336834 13665 ts_tablet_manager.cc:1918] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:48.336977 13665 log.cc:1196] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/3428d04e614e495e8853baf135a832d1
I20260924 03:10:48.337302 13665 ts_tablet_manager.cc:1939] T 3428d04e614e495e8853baf135a832d1 P 0e894773d44b4ab5bd762032231c2a99: Deleting consensus metadata
I20260924 03:10:48.337740 13120 catalog_manager.cc:4841] TS 0e894773d44b4ab5bd762032231c2a99 (127.0.98.3:45753): tablet 3428d04e614e495e8853baf135a832d1 (table test-table [id=0ce014cc5678461ba5c37cc621ff9bc6]) successfully deleted
W20260924 03:10:48.346443 13119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39711 (error 108)
I20260924 03:10:48.347118 13664 tablet_replica.cc:330] T 3428d04e614e495e8853baf135a832d1 P 8c0ae24c4b6546ef98e521624297d463: stopping tablet replica
I20260924 03:10:48.347309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13337
I20260924 03:10:48.358561 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.358428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39711 (local address 127.0.0.1:57622)
0924 03:10:48.358491 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.358526 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:10:48.358747 13119 catalog_manager.cc:4584] TS 9b515ed74c804cf287191149bd79ebcb (127.0.98.1:39711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3428d04e614e495e8853baf135a832d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39711: connect: Connection refused (error 111)
I20260924 03:10:48.367318 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:48.367218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39057 (local address 127.0.0.1:50170)
0924 03:10:48.367265 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:48.367289 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:10:48.370489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13470
I20260924 03:10:48.375295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13102
I20260924 03:10:48.380375 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:10:48.714354 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:10:48.714432 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:10:49.059073 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:10:49.059 UTC [12169] LOG: received smart shutdown request
2026-09-24 03:10:49.063 UTC [12169] LOG: background worker "logical replication launcher" (PID 12178) exited with exit code 1
2026-09-24 03:10:49.063 UTC [12173] LOG: shutting down
2026-09-24 03:10:49.202 UTC [12169] LOG: database system is shut down
2026-09-24T03:10:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44085 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:49 dist-test-slave-jtnz krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:49 dist-test-slave-jtnz krb5kdc[13678](info): set up 2 sockets
Sep 24 03:10:49 dist-test-slave-jtnz krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260924 03:10:50.273680 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:10:50.377745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:50.450897 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:50.451088 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:50.451120 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:50.452180 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:50.452244 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:50.452267 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:50.452296 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:50.452327 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:50.453784 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:50.454038 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:50.454252 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:50.458644 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.458662 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.458649 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:50.458870 13691 server_base.cc:1034] running on GCE node
I20260924 03:10:50.459120 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:50.459400 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:50.460541 13691 hybrid_clock.cc:648] HybridClock initialized: now 1790219450460541 us; error 40 us; skew 500 ppm
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:50.468307 13691 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:50.468654 13691 webserver.cc:460] Webserver started at http://127.0.98.62:45783/ using document root <none> and password file <none>
I20260924 03:10:50.468813 13691 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:50.468850 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:50.468928 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:50.469836 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "d46cd4aff50d4628894ebb3c536940ca"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "4d6c4b4bcfbca9a8b43efaba1e6d8785"
server_key_iv: "4737ee626df3938eba01796cb2d9afae"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.470116 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "d46cd4aff50d4628894ebb3c536940ca"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "4d6c4b4bcfbca9a8b43efaba1e6d8785"
server_key_iv: "4737ee626df3938eba01796cb2d9afae"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.471434 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:50.472107 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:50.472268 13691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:10:50.472369 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "d46cd4aff50d4628894ebb3c536940ca"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "4d6c4b4bcfbca9a8b43efaba1e6d8785"
server_key_iv: "4737ee626df3938eba01796cb2d9afae"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.472440 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:50.505210 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:50.506853 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:50.507020 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:50.510673 13691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46253
I20260924 03:10:50.510715 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46253 every 8 connection(s)
I20260924 03:10:50.511045 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:50.511830 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:50.512040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13691
I20260924 03:10:50.512143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:10:50.512329 392 external_mini_cluster.cc:1428] Setting key 67466161e59683829e14d0903447adaf
I20260924 03:10:50.514017 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:50.524948 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.514695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.0.98.62:46253)
0924 03:10:50.514882 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.514886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.514943 (+ 57us) server_negotiation.cc:407] Connection header received
0924 03:10:50.515420 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.515422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.515445 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.515536 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.515982 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.516656 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.517482 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.517766 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.517878 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.521763 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.521770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.521770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.521777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.523493 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.523857 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.523859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.523859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.523943 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.524156 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.524157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.524157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.524344 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.524431 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.524603 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.524729 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260924 03:10:50.524947 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.514674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46253 (local address 127.0.0.1:41232)
0924 03:10:50.514821 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.514857 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.515288 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.515542 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.515545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.515774 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.515895 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.515898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.516745 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.516746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.517375 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.517377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.517854 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.517854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.517921 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.518406 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.518413 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.521665 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.523586 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.523591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.523593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.523767 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.524009 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.524010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.524010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.524094 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.524411 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.524416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.524479 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.524580 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.524788 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":86,"thread_start_us":81,"threads_started":1}
I20260924 03:10:50.525740 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.526116 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.525991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55618)
0924 03:10:50.526024 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.526084 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 03:10:50.526270 13708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
I20260924 03:10:50.526770 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.526647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55634)
0924 03:10:50.526703 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.526732 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:10:50.528539 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34837: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:10:50.534852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34837
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:50.577760 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } attempt: 1
I20260924 03:10:50.578260 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.578114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55646)
0924 03:10:50.578190 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.578227 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:10:50.578591 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.578508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55662)
0924 03:10:50.578551 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.578574 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:10:50.578693 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34837: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
W20260924 03:10:50.612296 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:50.612504 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:50.612548 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:50.613646 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:50.613713 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:50.613735 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:50.613760 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:50.613788 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:50.615175 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:50.615433 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:50.615626 13766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:50.619874 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.619865 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.619915 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:50.620055 13766 server_base.cc:1034] running on GCE node
I20260924 03:10:50.620237 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:50.620468 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:50.621635 13766 hybrid_clock.cc:648] HybridClock initialized: now 1790219450621622 us; error 29 us; skew 500 ppm
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:50.623901 13766 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:10:50.624225 13766 webserver.cc:460] Webserver started at http://127.0.98.61:33535/ using document root <none> and password file <none>
I20260924 03:10:50.624370 13766 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:50.624403 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:50.624483 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:50.625294 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data/instance:
uuid: "4af6985fde55465c99eda2350dcea8e4"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "c08bf540c33e14f777bea526bf3d03c1"
server_key_iv: "63c6824fe8e2ff230b7a78276d93d078"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.625543 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal/instance:
uuid: "4af6985fde55465c99eda2350dcea8e4"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "c08bf540c33e14f777bea526bf3d03c1"
server_key_iv: "63c6824fe8e2ff230b7a78276d93d078"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.626645 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:10:50.627177 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:50.627326 13766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:10:50.627398 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "4af6985fde55465c99eda2350dcea8e4"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "c08bf540c33e14f777bea526bf3d03c1"
server_key_iv: "63c6824fe8e2ff230b7a78276d93d078"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.627444 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:50.641983 13766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:50.643651 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:50.643791 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:50.646836 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } attempt: 2
I20260924 03:10:50.647224 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.647100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55678)
0924 03:10:50.647166 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.647199 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:10:50.647598 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.647507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55680)
0924 03:10:50.647540 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.647571 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 03:10:50.647687 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34837: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34837: connect: Connection refused (error 111)
I20260924 03:10:50.647722 13766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34837
I20260924 03:10:50.647779 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34837 every 8 connection(s)
I20260924 03:10:50.648087 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:10:50.648906 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:50.650323 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.652307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13766
I20260924 03:10:50.652384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-1/wal/instance
I20260924 03:10:50.652572 392 external_mini_cluster.cc:1428] Setting key eaa1df6ae9143edd5d948f0c951729eb
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:50.659044 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.650928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.98.62:46253)
0924 03:10:50.650982 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.650989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.651195 (+ 206us) server_negotiation.cc:407] Connection header received
0924 03:10:50.651653 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.651654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.651668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.651730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.652296 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.652912 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.653857 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.654094 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.654171 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.656263 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.656265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.656265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.656273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.657832 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.658168 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.658170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.658170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.658247 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.658461 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.658468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.658469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.658598 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.658670 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.658878 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.658959 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:10:50.659058 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.650929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46253 (local address 127.0.0.1:41234)
0924 03:10:50.651087 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.651119 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.651557 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.651799 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.651802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.652032 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.652206 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.652209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.653000 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.653001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.653756 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.653758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.654176 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.654198 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.654318 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.654770 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.654778 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.656181 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.657905 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.657910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.657912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.658099 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.658316 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.658317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.658317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.658404 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.658664 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.658667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.658764 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.658834 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.658933 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 03:10:50.659633 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:10:50.666625 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.659839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55684)
0924 03:10:50.660023 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.660033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.660079 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.660226 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.660227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.660353 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.660390 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.660391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.661186 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.661187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.661689 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.661691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.662153 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.662153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.662237 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.662631 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.662636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.663925 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.665757 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.665757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.665758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.665898 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.666097 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.666098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.666098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.666150 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.666450 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.666451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.666480 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.666523 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.666579 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 03:10:50.666760 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.659892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.98.61:34837)
0924 03:10:50.660029 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.660034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.660076 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:10:50.660131 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.660133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.660144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.660223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.660474 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.661100 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.661783 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.662066 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.662152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.664001 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.664008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.664008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.664015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.665676 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.665974 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.665977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.665977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.666035 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.666235 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.666236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.666236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.666386 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.666474 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.666556 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.666634 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
I20260924 03:10:50.667315 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.667616 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.667503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55308)
0924 03:10:50.667548 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.667591 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:10:50.667739 13783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
I20260924 03:10:50.668147 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.668046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55324)
0924 03:10:50.668094 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.668122 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 03:10:50.668277 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:10:50.675056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45585
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:50.717500 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } attempt: 1
I20260924 03:10:50.717907 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.717802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55340)
0924 03:10:50.717851 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.717879 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:10:50.718261 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.718169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55352)
0924 03:10:50.718222 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.718242 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 03:10:50.718387 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
I20260924 03:10:50.738821 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } attempt: 3
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:10:50.745235 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.739204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.98.61:34837)
0924 03:10:50.739262 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.739265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.739276 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:10:50.739363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.739364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.739374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.739427 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.739754 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.740257 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.740901 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.741091 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.741163 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.742946 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.742948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.742948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.742954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.744217 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.744529 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.744530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.744531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.744584 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.744765 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.744765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.744765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.744856 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.744920 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.745078 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.745147 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:10:50.745231 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.739176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55696)
0924 03:10:50.739238 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.739247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.739285 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.739440 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.739441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.739605 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.739667 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.739669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.740339 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.740340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.740812 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.740814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.741160 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.741161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.741234 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.741707 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.741713 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.742864 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.744307 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.744309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.744309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.744466 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.744650 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.744651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.744651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.744703 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.744913 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.744914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.744960 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.745058 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.745145 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 03:10:50.745903 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.746243 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.746123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55366)
0924 03:10:50.746166 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.746218 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:10:50.746572 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.746494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55368)
0924 03:10:50.746530 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.746551 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 03:10:50.746657 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
W20260924 03:10:50.763609 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:50.763774 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:50.763794 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:50.764894 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:50.764931 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:50.764943 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:50.764994 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:50.765014 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:50.766535 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46253,127.0.98.61:34837,127.0.98.60:45585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:50.766762 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:50.766978 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:50.771191 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.771224 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:50.771245 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:50.771466 13841 server_base.cc:1034] running on GCE node
I20260924 03:10:50.771785 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:50.772001 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:50.773166 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790219450773157 us; error 38 us; skew 500 ppm
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:50.775424 13841 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:10:50.775766 13841 webserver.cc:460] Webserver started at http://127.0.98.60:42643/ using document root <none> and password file <none>
I20260924 03:10:50.775981 13841 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:50.776024 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:50.776108 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:50.776994 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data/instance:
uuid: "44b360dca2d2425dbda9c94a15ba9765"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "fdf0cb90cedd3860da768495cd1cb9e7"
server_key_iv: "ac1c86fb1c60986a9695eb8687eabba8"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.777262 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal/instance:
uuid: "44b360dca2d2425dbda9c94a15ba9765"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "fdf0cb90cedd3860da768495cd1cb9e7"
server_key_iv: "ac1c86fb1c60986a9695eb8687eabba8"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.777772 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } attempt: 1
I20260924 03:10:50.778081 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.777999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55382)
0924 03:10:50.778034 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.778061 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 03:10:50.778468 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.778387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55396)
0924 03:10:50.778428 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.778450 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:10:50.778618 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
I20260924 03:10:50.778787 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:10:50.779428 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:50.779597 13841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:10:50.779686 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "44b360dca2d2425dbda9c94a15ba9765"
format_stamp: "Formatted at 2026-09-24 03:10:50 on dist-test-slave-jtnz"
server_key: "fdf0cb90cedd3860da768495cd1cb9e7"
server_key_iv: "ac1c86fb1c60986a9695eb8687eabba8"
server_key_version: "encryptionkey@0"
I20260924 03:10:50.779767 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:50.786530 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } attempt: 2
I20260924 03:10:50.786847 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.786745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55402)
0924 03:10:50.786796 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.786826 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 03:10:50.787158 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.787071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55406)
0924 03:10:50.787113 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.787135 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:10:50.787237 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45585: connect: Connection refused (error 111)
I20260924 03:10:50.790880 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:50.792459 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:50.792613 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:50.796180 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45585
I20260924 03:10:50.796208 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45585 every 8 connection(s)
I20260924 03:10:50.796504 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:10:50.797256 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:50.798596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13841
I20260924 03:10:50.798699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219310260848-392-0/minicluster-data/master-2/wal/instance
I20260924 03:10:50.798902 392 external_mini_cluster.cc:1428] Setting key d7dae1bae4f7124af05caebfe73693cd
I20260924 03:10:50.798805 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:10:50.807549 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.799668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.98.62:46253)
0924 03:10:50.799708 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.799710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.799843 (+ 133us) server_negotiation.cc:407] Connection header received
0924 03:10:50.800167 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.800168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.800179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.800247 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.800718 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.801182 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.802258 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.802562 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.802626 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.804601 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.804602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.804603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.804610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.806151 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.806649 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.806650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.806650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.806700 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.806949 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.806950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.806950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.807040 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.807121 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.807379 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.807456 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:10:50.807688 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.799596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46253 (local address 127.0.0.1:41244)
0924 03:10:50.799730 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.799764 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.800067 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.800267 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.800269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.800499 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.800635 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.800638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.801252 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.801253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.802111 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.802113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.802641 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.802642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.802773 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.803206 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.803221 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.804513 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.806244 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.806250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.806253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.806577 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.806766 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.806767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.806767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.806885 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.807115 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.807121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.807249 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.807345 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.807475 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260924 03:10:50.808152 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.808315 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.799359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.0.98.60:45585)
0924 03:10:50.799523 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.799532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.799554 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:10:50.800064 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.800073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.800102 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.800185 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.800793 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.801713 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.802633 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.803046 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.803132 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.805290 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.805297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.805297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.805304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.807017 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.807324 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.807327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.807328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.807384 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.807577 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.807578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.807579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.807726 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.807863 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.808078 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.808146 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:50.814746 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.808438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34837 (local address 127.0.0.1:55710)
0924 03:10:50.808530 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.808539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.808574 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.808706 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.808706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.808842 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.808882 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.808884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.809517 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.809518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.810084 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.810086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.810491 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.810492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.810553 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.810858 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.810862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.812102 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.813713 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.813714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.813715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.813876 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.814092 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.814093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.814094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.814151 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.814441 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.814442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.814485 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.814582 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.814665 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:10:50.814787 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.808530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.98.61:34837)
0924 03:10:50.808583 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.808585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.808595 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:10:50.808635 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.808636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.808645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.808705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.808946 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.809445 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.810158 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.810418 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.810466 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.812186 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.812187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.812187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.812194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.813638 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.813951 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.813952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.813953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.814003 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.814258 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.814259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.814260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.814368 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.814441 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.814609 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.814676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:10:50.815182 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:10:50.816488 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.809111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.62:46253)
0924 03:10:50.809147 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.809148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.809157 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:50.809206 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.809207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.809215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.809266 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.809747 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.810170 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.810978 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.811137 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.811186 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.813433 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.813434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.813434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.813439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.815044 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.815376 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.815377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.815377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.815429 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.815644 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.815645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.815645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.815782 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.816317 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.816349 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.816403 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:10:50.821753 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.815407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55422)
0924 03:10:50.815493 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.815502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.815541 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.815656 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.815656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.815800 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.815831 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.815833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.816476 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.816477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.817289 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.817292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.817776 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.817776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.817823 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.818146 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.818151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.819233 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.820823 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.820824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.820824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.821043 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.821222 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.821223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.821223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.821280 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.821565 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.821565 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.821601 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.821647 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.821703 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:10:50.821867 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.815452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.0.98.60:45585)
0924 03:10:50.815495 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.815496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.815531 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:10:50.815589 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.815590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.815599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.815645 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.815895 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.816405 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.817351 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.817695 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.817766 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.819305 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.819306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.819307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.819312 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.820747 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.821112 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.821113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.821113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.821156 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.821348 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.821349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.821349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.821474 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.821544 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.821702 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.821771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:10:50.823355 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: Bootstrap starting.
I20260924 03:10:50.824249 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.817068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.98.61:34837)
0924 03:10:50.817097 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.817098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.817106 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:10:50.817152 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.817152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.817159 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.817219 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.817775 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.818469 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.819101 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.819374 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.819420 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.821924 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.821925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.821925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.821932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.823292 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.823642 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.823643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.823643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.823685 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.823871 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.823872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.823872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.823950 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.824057 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.824142 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.824192 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:10:50.824332 13909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:50.824656 13909 log.cc:824] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:50.825305 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: No bootstrap required, opened a new log
I20260924 03:10:50.825773 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:10:50.827104 13909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.827232 13909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:50.827273 13909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b360dca2d2425dbda9c94a15ba9765, State: Initialized, Role: FOLLOWER
I20260924 03:10:50.827378 13909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.827754 13909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:50.827735 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:50.827891 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:10:50.829993 13841 master_runner.cc:186] Error getting master registration for 127.0.98.62:46253: OK, instance_id { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" instance_seqno: 1790219450509654 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:10:50.832875 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 03:10:50.829334 13933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:10:50.833058 13933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:10:50.833918 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.826938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.0.98.62:46253)
0924 03:10:50.826967 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.826969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.827402 (+ 433us) server_negotiation.cc:407] Connection header received
0924 03:10:50.827465 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.827466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.827473 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.827518 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.828061 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.828548 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.829465 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.829723 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.829773 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.831288 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.831289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.831289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.831295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.832877 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.833259 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.833260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.833260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.833309 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.833483 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.833484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.833484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.833586 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.833659 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.833784 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.833848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:10:50.835129 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.826988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.98.61:34837)
0924 03:10:50.827025 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.827037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.827140 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:10:50.827197 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.827197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.827205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.827256 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.827891 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.828475 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.829347 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.829659 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.829698 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.831422 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.831423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.831424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.831430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.833823 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.834431 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.834432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.834433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.834481 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.834663 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.834664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.834664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.834774 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.834858 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.834956 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.835034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:10:50.835987 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.827168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.98.60:45585)
0924 03:10:50.827195 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.827196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.829127 (+ 1931us) server_negotiation.cc:407] Connection header received
0924 03:10:50.829197 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.829197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.829204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.829226 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.829860 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.830545 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.831559 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.831849 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.831891 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.833366 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.833367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.833368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.833373 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.834914 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.835264 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.835265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.835265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.835314 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.835515 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.835515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.835516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.835641 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.835698 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.835884 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.835936 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 03:10:50.853749 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } attempt: 2
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:10:50.859665 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.854051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.60:45585)
0924 03:10:50.854105 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.854106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.854113 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:50.854128 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.854129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.854135 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.854206 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.854443 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.854813 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.855385 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.855547 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.855619 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.857295 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.857296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.857296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.857302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.858748 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.859097 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.859098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.859098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.859143 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.859316 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.859317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.859317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.859390 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.859431 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.859570 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.859617 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:10:50.859719 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.853968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55440)
0924 03:10:50.854016 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.854025 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.854065 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.854209 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.854210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.854342 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.854369 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.854370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.854882 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.854883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.855300 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.855301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.855621 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.855622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.855669 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.856055 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.856058 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.857211 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.858821 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.858822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.858822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.859030 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.859204 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.859204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.859205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.859262 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.859445 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.859445 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.859485 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.859556 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.859633 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:10:50.860797 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: Bootstrap starting.
I20260924 03:10:50.861339 13759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:50.861613 13759 log.cc:824] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:50.862324 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: No bootstrap required, opened a new log
I20260924 03:10:50.863655 13759 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.863768 13759 raft_consensus.cc:374] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:50.863811 13759 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46cd4aff50d4628894ebb3c536940ca, State: Initialized, Role: FOLLOWER
I20260924 03:10:50.863935 13759 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.864245 13759 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:50.864534 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:50.864617 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:10:50.865860 13947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:10:50.865911 13947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:10:50.866577 13691 master_runner.cc:186] Error getting master registration for 127.0.98.61:34837: OK, instance_id { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" instance_seqno: 1790219450646387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:10:50.868450 13691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:10:50.878355 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } attempt: 3
Sep 24 03:10:50 dist-test-slave-jtnz krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:10:50.884341 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.878569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45585 (local address 127.0.0.1:55442)
0924 03:10:50.878608 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:50.878617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:50.878651 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:50.878771 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:50.878772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:50.878894 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:50.878926 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.878927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.879492 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.879493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.879959 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:50.879961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.880296 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.880297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:50.880355 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.880749 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:50.880754 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:50.881852 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:50.883396 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.883397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.883398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.883561 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.883761 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:50.883761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:50.883762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:50.883814 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:50.884070 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:50.884071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:50.884110 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:10:50.884208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:10:50.884285 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 03:10:50.884677 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:50.878592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.0.98.60:45585)
0924 03:10:50.878636 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:50.878637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:50.878644 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:50.878701 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:50.878701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:50.878708 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:50.878767 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:50.879024 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.879408 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.880044 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:50.880218 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:50.880295 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:50.881957 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:50.881958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:50.881959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:50.881972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:50.883322 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.883641 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.883642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.883642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.883692 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:50.883896 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:50.883897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:50.883897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:50.883998 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:50.884065 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:50.884533 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 03:10:50.884599 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:10:50.885926 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: Bootstrap starting.
I20260924 03:10:50.886492 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:50.886739 13834 log.cc:824] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:50.887458 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: No bootstrap required, opened a new log
I20260924 03:10:50.888674 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.888787 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:50.888826 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af6985fde55465c99eda2350dcea8e4, State: Initialized, Role: FOLLOWER
I20260924 03:10:50.888937 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:50.889178 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:50.889230 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:50.889297 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:10:50.890910 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:10:50.891009 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:10:50.892797 13766 master_runner.cc:263] No leader master found from master 4af6985fde55465c99eda2350dcea8e4
I20260924 03:10:50.892838 13766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 03:10:50.894292 13787 tablet.cc:2369] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:51.007869 13712 tablet.cc:2369] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:10:51.043124 13862 tablet.cc:2369] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:10:51.176267 13936 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:51.176367 13936 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:51.176628 13936 leader_election.cc:288] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af6985fde55465c99eda2350dcea8e4 (127.0.98.61:34837), 44b360dca2d2425dbda9c94a15ba9765 (127.0.98.60:45585)
I20260924 03:10:51.176911 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46cd4aff50d4628894ebb3c536940ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b360dca2d2425dbda9c94a15ba9765" is_pre_election: true
I20260924 03:10:51.176914 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46cd4aff50d4628894ebb3c536940ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af6985fde55465c99eda2350dcea8e4" is_pre_election: true
I20260924 03:10:51.177037 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46cd4aff50d4628894ebb3c536940ca in term 0.
I20260924 03:10:51.177065 13808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d46cd4aff50d4628894ebb3c536940ca in term 0.
I20260924 03:10:51.177255 13708 leader_election.cc:302] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af6985fde55465c99eda2350dcea8e4, d46cd4aff50d4628894ebb3c536940ca; no voters:
I20260924 03:10:51.177420 13936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:51.177495 13936 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:51.177512 13936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:51.178242 13936 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:51.178363 13936 leader_election.cc:288] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [CANDIDATE]: Term 1 election: Requested vote from peers 4af6985fde55465c99eda2350dcea8e4 (127.0.98.61:34837), 44b360dca2d2425dbda9c94a15ba9765 (127.0.98.60:45585)
I20260924 03:10:51.178573 13808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46cd4aff50d4628894ebb3c536940ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af6985fde55465c99eda2350dcea8e4"
I20260924 03:10:51.178594 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46cd4aff50d4628894ebb3c536940ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b360dca2d2425dbda9c94a15ba9765"
I20260924 03:10:51.178648 13808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:51.178649 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:51.179302 13808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46cd4aff50d4628894ebb3c536940ca in term 1.
I20260924 03:10:51.179307 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46cd4aff50d4628894ebb3c536940ca in term 1.
I20260924 03:10:51.179492 13708 leader_election.cc:302] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b360dca2d2425dbda9c94a15ba9765, d46cd4aff50d4628894ebb3c536940ca; no voters:
I20260924 03:10:51.179587 13936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:51.179695 13936 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [term 1 LEADER]: Becoming Leader. State: Replica: d46cd4aff50d4628894ebb3c536940ca, State: Running, Role: LEADER
I20260924 03:10:51.179760 13936 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } }
I20260924 03:10:51.180132 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46cd4aff50d4628894ebb3c536940ca. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.180200 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:51.180377 13965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:51.180522 13965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:51.181743 13808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [term 1 FOLLOWER]: Refusing update from remote peer d46cd4aff50d4628894ebb3c536940ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:51.181782 13883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [term 1 FOLLOWER]: Refusing update from remote peer d46cd4aff50d4628894ebb3c536940ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:51.182034 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:51.182133 13936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:51.183248 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46cd4aff50d4628894ebb3c536940ca. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.183271 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46cd4aff50d4628894ebb3c536940ca. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.183337 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:10:51.183355 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:10:51.184327 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.184414 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:51.184475 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.184484 13967 mvcc.cc:205] Tried to move back new op lower bound from 7332738872038158336 to 7332738872032722944. Current Snapshot: MvccSnapshot[applied={T|T < 7332738872038158336}]
I20260924 03:10:51.184525 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:51.184491 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.184566 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:10:51.184607 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46cd4aff50d4628894ebb3c536940ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46cd4aff50d4628894ebb3c536940ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46253 } } peers { permanent_uuid: "4af6985fde55465c99eda2350dcea8e4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34837 } } peers { permanent_uuid: "44b360dca2d2425dbda9c94a15ba9765" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45585 } } }
I20260924 03:10:51.184664 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:10:51.184739 13965 catalog_manager.cc:1347] Generated new cluster ID: 33409ba40b224b6aa775e1fce2678d5b
I20260924 03:10:51.184734 13968 mvcc.cc:205] Tried to move back new op lower bound from 7332738872038158336 to 7332738872032722944. Current Snapshot: MvccSnapshot[applied={T|T < 7332738872038158336}]
I20260924 03:10:51.184796 13965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:51.191656 13965 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:51.192148 13965 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:51.197815 13965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46cd4aff50d4628894ebb3c536940ca: Generated new TSK 0
I20260924 03:10:51.197983 13965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:10:51.833491 13933 catalog_manager.cc:1259] Loaded cluster ID: 33409ba40b224b6aa775e1fce2678d5b
I20260924 03:10:51.833583 13933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: loading cluster ID for follower catalog manager: success
I20260924 03:10:51.834355 13933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: acquiring CA information for follower catalog manager: success
I20260924 03:10:51.834601 13933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44b360dca2d2425dbda9c94a15ba9765: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:10:51.891443 13961 catalog_manager.cc:1259] Loaded cluster ID: 33409ba40b224b6aa775e1fce2678d5b
I20260924 03:10:51.891502 13961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: loading cluster ID for follower catalog manager: success
I20260924 03:10:51.892400 13961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: acquiring CA information for follower catalog manager: success
I20260924 03:10:51.892697 13961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4af6985fde55465c99eda2350dcea8e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:10:52.250898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13691
I20260924 03:10:52.255574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13766
I20260924 03:10:52.259995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13841
2026-09-24T03:10:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2949 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:52 dist-test-slave-jtnz krb5kdc[13981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:52 dist-test-slave-jtnz krb5kdc[13981](info): set up 2 sockets
Sep 24 03:10:52 dist-test-slave-jtnz krb5kdc[13981](info): commencing operation
krb5kdc: starting...
W20260924 03:10:53.217824 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:10:53.319271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35469
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:53.391074 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:53.391269 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:53.391304 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:53.392407 13994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:53.392465 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:53.392488 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:53.392513 13994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:53.392545 13994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:53.393898 13994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:53.394121 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:53.394328 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:53.398391 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:53.398412 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:53.398454 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:53.398422 13994 server_base.cc:1034] running on GCE node
I20260924 03:10:53.398831 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:53.399086 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:53.400215 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790219453400204 us; error 25 us; skew 500 ppm
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:53.402415 13994 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:53.402853 13994 webserver.cc:460] Webserver started at http://127.0.98.62:40703/ using document root <none> and password file <none>
I20260924 03:10:53.403064 13994 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:53.403127 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:53.403251 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:53.404134 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "de8e6f72b8a04d05bd96c8911ba32a06"
format_stamp: "Formatted at 2026-09-24 03:10:53 on dist-test-slave-jtnz"
server_key: "983bfd41e24353f314f70388d98018c4"
server_key_iv: "b5a126f360902d27b0967873dca9667b"
server_key_version: "encryptionkey@0"
I20260924 03:10:53.404441 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "de8e6f72b8a04d05bd96c8911ba32a06"
format_stamp: "Formatted at 2026-09-24 03:10:53 on dist-test-slave-jtnz"
server_key: "983bfd41e24353f314f70388d98018c4"
server_key_iv: "b5a126f360902d27b0967873dca9667b"
server_key_version: "encryptionkey@0"
I20260924 03:10:53.405550 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:10:53.406148 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:53.406322 13994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:53.406406 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "de8e6f72b8a04d05bd96c8911ba32a06"
format_stamp: "Formatted at 2026-09-24 03:10:53 on dist-test-slave-jtnz"
server_key: "983bfd41e24353f314f70388d98018c4"
server_key_iv: "b5a126f360902d27b0967873dca9667b"
server_key_version: "encryptionkey@0"
I20260924 03:10:53.406481 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:53.415911 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:53.417282 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:53.417418 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:53.421037 13994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35469
I20260924 03:10:53.421064 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35469 every 8 connection(s)
I20260924 03:10:53.421346 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:53.422068 14062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:53.422686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 13994
I20260924 03:10:53.422782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:10:53.422966 392 external_mini_cluster.cc:1428] Setting key b211d76bc86979d93edd29a2f3aa32ee
I20260924 03:10:53.424778 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06: Bootstrap starting.
I20260924 03:10:53.425366 14062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:53.425623 14062 log.cc:824] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:53.426306 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06: No bootstrap required, opened a new log
I20260924 03:10:53.428532 14062 raft_consensus.cc:348] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } }
I20260924 03:10:53.428687 14062 raft_consensus.cc:374] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:53.428762 14062 raft_consensus.cc:729] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6f72b8a04d05bd96c8911ba32a06, State: Initialized, Role: FOLLOWER
I20260924 03:10:53.428910 14062 consensus_queue.cc:260] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } }
I20260924 03:10:53.428997 14062 raft_consensus.cc:388] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:53.429039 14062 raft_consensus.cc:482] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:53.429095 14062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:53.429605 14062 raft_consensus.cc:504] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } }
I20260924 03:10:53.429698 14062 leader_election.cc:302] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de8e6f72b8a04d05bd96c8911ba32a06; no voters:
I20260924 03:10:53.429840 14062 leader_election.cc:288] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:53.429908 14067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:53.430037 14067 raft_consensus.cc:686] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [term 1 LEADER]: Becoming Leader. State: Replica: de8e6f72b8a04d05bd96c8911ba32a06, State: Running, Role: LEADER
I20260924 03:10:53.430110 14062 sys_catalog.cc:564] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:53.430119 14067 consensus_queue.cc:237] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } }
I20260924 03:10:53.430714 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de8e6f72b8a04d05bd96c8911ba32a06. Latest consensus state: current_term: 1 leader_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } } }
I20260924 03:10:53.430869 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8e6f72b8a04d05bd96c8911ba32a06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35469 } } }
I20260924 03:10:53.430933 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:53.431005 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:53.431325 14077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:53.431854 14077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:53.433022 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:53.423418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.98.62:35469)
0924 03:10:53.423567 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:53.423573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:53.423594 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:10:53.423940 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:53.423947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:53.423983 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:53.424082 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:53.424812 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:53.425756 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:53.426717 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:53.427059 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:53.427131 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:53.429643 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:53.429646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:53.429646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:53.429657 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:53.431521 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:53.431909 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:53.431911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:53.431911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:53.432006 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:53.432224 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:53.432225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:53.432225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:53.432501 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:53.432620 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:53.432726 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 03:10:53.432846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 03:10:53.433437 14077 catalog_manager.cc:1347] Generated new cluster ID: 9d8fb7be6c02403a8b1d19aa68ce66aa
I20260924 03:10:53.433502 14077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:53.433805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:10:53.463778 14077 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:53.464255 14077 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:53.472499 14077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de8e6f72b8a04d05bd96c8911ba32a06: Generated new TSK 0
I20260924 03:10:53.472666 14077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:10:53.487183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 13994
2026-09-24T03:10:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1225 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[14089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[14089](info): set up 2 sockets
Sep 24 03:10:53 dist-test-slave-jtnz krb5kdc[14089](info): commencing operation
krb5kdc: starting...
W20260924 03:10:54.454090 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:10:54 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:10:54.560695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:54.632238 14102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:54.632416 14102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:54.632437 14102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:54.633517 14102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:54.633553 14102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:54.633565 14102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:54.633615 14102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:54.633643 14102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:54.634930 14102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:54.635149 14102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:54.635345 14102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:54.639680 14108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:54.639655 14110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:54.639672 14107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:54.639994 14102 server_base.cc:1034] running on GCE node
I20260924 03:10:54.640189 14102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:54.640384 14102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:54.641515 14102 hybrid_clock.cc:648] HybridClock initialized: now 1790219454641501 us; error 26 us; skew 500 ppm
Sep 24 03:10:54 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:54.643821 14102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:54.644232 14102 webserver.cc:460] Webserver started at http://127.0.98.62:44473/ using document root <none> and password file <none>
I20260924 03:10:54.644392 14102 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:54.644424 14102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:54.644495 14102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:54.645262 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "693631d2f02b49ef9682317ba512eee6"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "98c1ce7ff2e8d7465e62741885bb26c6"
server_key_iv: "29216c37b8bb7dd631db0d182f85d4df"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.645520 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "693631d2f02b49ef9682317ba512eee6"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "98c1ce7ff2e8d7465e62741885bb26c6"
server_key_iv: "29216c37b8bb7dd631db0d182f85d4df"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.646746 14102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:10:54.647289 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:54.647482 14102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:54.647547 14102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "693631d2f02b49ef9682317ba512eee6"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "98c1ce7ff2e8d7465e62741885bb26c6"
server_key_iv: "29216c37b8bb7dd631db0d182f85d4df"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.647635 14102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:54.677456 14102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:54.679076 14102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:54.679216 14102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:54.682752 14102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40187
I20260924 03:10:54.682806 14169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40187 every 8 connection(s)
I20260924 03:10:54.683104 14102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:54.683969 14170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:54.684464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14102
I20260924 03:10:54.684540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:10:54.684717 392 external_mini_cluster.cc:1428] Setting key b2ebe455d8c2fd6c74485e32af910cec
I20260924 03:10:54.686645 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap starting.
I20260924 03:10:54.687198 14170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:54.687500 14170 log.cc:824] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:54.688241 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: No bootstrap required, opened a new log
I20260924 03:10:54.690496 14170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:54.690649 14170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:54.690672 14170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Initialized, Role: FOLLOWER
I20260924 03:10:54.690795 14170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:54.690852 14170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:54.690871 14170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:54.690958 14170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:10:54 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:54.691454 14170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:54.691557 14170 leader_election.cc:302] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693631d2f02b49ef9682317ba512eee6; no voters:
I20260924 03:10:54.691708 14170 leader_election.cc:288] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:10:54.691744 14175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:54.691883 14175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 LEADER]: Becoming Leader. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Running, Role: LEADER
I20260924 03:10:54.691937 14170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:54.691960 14175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:54.692674 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:54.692752 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:54.693094 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693631d2f02b49ef9682317ba512eee6. Latest consensus state: current_term: 1 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:54.693167 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:54.693495 14184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:54.693908 14184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:54.695286 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:54.685166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.98.62:40187)
0924 03:10:54.685325 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:54.685332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:54.685354 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:10:54.685698 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:54.685704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:54.685742 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:54.685847 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:54.686676 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.687623 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.688594 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.688985 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.689058 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:54.691598 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:54.691602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:54.691602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:54.691612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:54.693783 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:54.694162 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:54.694165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:54.694165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:54.694274 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:54.694480 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:54.694481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:54.694482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:54.694741 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:54.694874 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:54.695087 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:10:54.695174 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260924 03:10:54.695809 14184 catalog_manager.cc:1347] Generated new cluster ID: 2f2ae83ec4444415864ec77c47ac4aef
I20260924 03:10:54.695852 14184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:54.714293 14184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:10:54.714742 14184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:54.722193 14184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Generated new TSK 0
I20260924 03:10:54.722358 14184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:10:54.753075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:54.841804 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:54.841984 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:54.842007 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:54.843071 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:54.843106 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:54.843129 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:54.844403 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:54.844614 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:54.844839 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:54.849409 14204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:54.849429 14202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:54.849417 14201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:54.849431 14196 server_base.cc:1034] running on GCE node
I20260924 03:10:54.849736 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:54.849953 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:54.851120 14196 hybrid_clock.cc:648] HybridClock initialized: now 1790219454851100 us; error 38 us; skew 500 ppm
Sep 24 03:10:54 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:54.853356 14196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:54.853729 14196 webserver.cc:460] Webserver started at http://127.0.98.1:46789/ using document root <none> and password file <none>
I20260924 03:10:54.853952 14196 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:54.854015 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:54.854158 14196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:54.855108 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2d305973b214c04b11464e1ca841e97"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "5539e345f11b8e1c3c3886b57ba5cdfb"
server_key_iv: "2fc99e000251af7a212b239d8930996f"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.855440 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d305973b214c04b11464e1ca841e97"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "5539e345f11b8e1c3c3886b57ba5cdfb"
server_key_iv: "2fc99e000251af7a212b239d8930996f"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.856794 14196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:10:54.857457 14211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:54.857622 14196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:10:54.857709 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "a2d305973b214c04b11464e1ca841e97"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "5539e345f11b8e1c3c3886b57ba5cdfb"
server_key_iv: "2fc99e000251af7a212b239d8930996f"
server_key_version: "encryptionkey@0"
I20260924 03:10:54.857800 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:54.881331 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:54.883358 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:54.883536 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:54.883797 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:54.884124 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:54.884179 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:54.884222 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:54.884248 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:54.889907 14196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36513
I20260924 03:10:54.889945 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36513 every 8 connection(s)
I20260924 03:10:54.890308 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:10:54 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:54.898066 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:54.890940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56155 (local address 127.0.98.62:40187)
0924 03:10:54.890986 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:54.890989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:54.891218 (+ 229us) server_negotiation.cc:407] Connection header received
0924 03:10:54.891636 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:54.891637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:54.891646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:54.891693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:54.892590 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.893252 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.894093 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.894343 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.894391 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:54.896064 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:54.896065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:54.896066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:54.896072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:54.897074 (+ 1002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:54.897372 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:54.897372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:54.897372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:54.897413 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:54.897592 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:54.897592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:54.897592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:54.897693 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:54.897737 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:54.897946 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 03:10:54.898010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:10:54.898213 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:54.890950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:56155)
0924 03:10:54.891093 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:54.891127 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:54.891563 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:54.891739 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:54.891742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:54.892314 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:54.892512 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:54.892517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.893321 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.893322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:54.894028 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:54.894030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:54.894403 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:54.894404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:54.894490 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:54.894826 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:54.894832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:54.895991 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:54.897134 (+ 1143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:54.897140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:54.897141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:54.897316 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:54.897463 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:54.897463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:54.897463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:54.897545 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:54.897741 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:54.897744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:54.897839 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 03:10:54.897915 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:10:54.898013 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":86,"threads_started":1}
I20260924 03:10:54.898671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14196
I20260924 03:10:54.898773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:10:54.898816 14325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:54.898895 14325 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:54.898931 392 external_mini_cluster.cc:1428] Setting key 7f13c96fdb31a4361612ac9f518fe7d1
I20260924 03:10:54.899008 14325 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:54.899477 14133 ts_manager.cc:194] Registered new tserver with Master: a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513)
I20260924 03:10:54.899973 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56155
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:10:54.920915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:54.991976 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:54.992266 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:54.992295 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:54.993396 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:54.993429 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:54.993453 14330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:54.994678 14330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:54.994880 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:54.995067 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:54.999181 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:54.999185 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:54.999264 14330 server_base.cc:1034] running on GCE node
W20260924 03:10:54.999185 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:54.999560 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:54.999792 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.000926 14330 hybrid_clock.cc:648] HybridClock initialized: now 1790219455000912 us; error 24 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.003396 14330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:55.003793 14330 webserver.cc:460] Webserver started at http://127.0.98.2:40417/ using document root <none> and password file <none>
I20260924 03:10:55.004011 14330 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.004077 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.004244 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:55.005193 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "8617d4955b2d260a0f379ee596db80f9"
server_key_iv: "1268c2150ebd176c9ce262bbf0918b6e"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.005537 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "8617d4955b2d260a0f379ee596db80f9"
server_key_iv: "1268c2150ebd176c9ce262bbf0918b6e"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.006808 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:10:55.007465 14345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.007658 14330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:55.007758 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "8617d4955b2d260a0f379ee596db80f9"
server_key_iv: "1268c2150ebd176c9ce262bbf0918b6e"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.007853 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.019313 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.021322 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.021485 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.021680 14330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:55.021947 14330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:55.021998 14330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.022037 14330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:55.022063 14330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.027370 14330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45833
I20260924 03:10:55.027432 14458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45833 every 8 connection(s)
I20260924 03:10:55.027707 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:55.034624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14330
I20260924 03:10:55.034729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:10:55.034935 392 external_mini_cluster.cc:1428] Setting key ac3dfebf71070c20251db4cfbcf1aad3
I20260924 03:10:55.036667 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.028306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58443 (local address 127.0.98.62:40187)
0924 03:10:55.028395 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.028398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.028555 (+ 157us) server_negotiation.cc:407] Connection header received
0924 03:10:55.029013 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.029015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.029027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.029108 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.030017 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.030593 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.031517 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.031732 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.031808 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.033749 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.033750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.033751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.033758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.035312 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.035704 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.035705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.035705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.035784 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.036012 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.036013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.036013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.036152 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.036221 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.036429 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.036534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:10:55.036736 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.028281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:58443)
0924 03:10:55.028440 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.028475 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.028913 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.029174 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.029177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.029742 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.029915 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.029919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.030680 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.030681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.031428 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.031430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.031817 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.031818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.031961 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.032431 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.032438 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.033656 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.035403 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.035409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.035411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.035634 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.035848 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.035849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.035849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.035950 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.036214 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.036217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.036313 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.036389 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.036491 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 03:10:55.037182 14459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:55.037285 14459 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:55.037469 14459 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:55.037818 14133 ts_manager.cc:194] Registered new tserver with Master: 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833)
I20260924 03:10:55.038229 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58443
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:10:55.056806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:55.129655 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:55.129827 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:55.129849 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:55.131033 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:55.131075 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:55.131122 14464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:55.132501 14464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:55.132709 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:55.132910 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:55.137388 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.137396 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.137396 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.137604 14464 server_base.cc:1034] running on GCE node
I20260924 03:10:55.137844 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:55.138079 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.139221 14464 hybrid_clock.cc:648] HybridClock initialized: now 1790219455139210 us; error 22 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.141569 14464 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:55.141979 14464 webserver.cc:460] Webserver started at http://127.0.98.3:40503/ using document root <none> and password file <none>
I20260924 03:10:55.142144 14464 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.142241 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.142333 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:10:55.143227 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "b68866397240470fad4d46f6e128a4d4"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "632331cafd3d781719c8bc075319f807"
server_key_iv: "3df4616f40306a5cd8edb80d3218011d"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.143515 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b68866397240470fad4d46f6e128a4d4"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "632331cafd3d781719c8bc075319f807"
server_key_iv: "3df4616f40306a5cd8edb80d3218011d"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.144600 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:10:55.145215 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.145370 14464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:55.145447 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "b68866397240470fad4d46f6e128a4d4"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "632331cafd3d781719c8bc075319f807"
server_key_iv: "3df4616f40306a5cd8edb80d3218011d"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.145514 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.156839 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.158998 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.159142 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.159336 14464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:55.159603 14464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:55.159634 14464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.159653 14464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:55.159695 14464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.165184 14464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260924 03:10:55.165236 14592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260924 03:10:55.165495 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:55.170780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14464
I20260924 03:10:55.170928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:10:55.171121 392 external_mini_cluster.cc:1428] Setting key 49091be0d717523d33e2962d7933d22d
I20260924 03:10:55.174080 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.166141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53361 (local address 127.0.98.62:40187)
0924 03:10:55.166246 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.166249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.166414 (+ 165us) server_negotiation.cc:407] Connection header received
0924 03:10:55.166840 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.166841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.166850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.166901 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.167730 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.168189 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.169057 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.169228 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.169274 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.171320 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.171322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.171322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.171329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.172755 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.173160 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.173161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.173162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.173215 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.173481 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.173482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.173483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.173596 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.173671 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.173919 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.173990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:10:55.174314 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.166105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:53361)
0924 03:10:55.166287 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.166330 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.166762 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.166949 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.166952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.167498 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.167662 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.167666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.168259 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.168259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.168991 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.168993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.169291 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.169292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.169378 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.169776 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.169783 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.171223 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.172856 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.172863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.172865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.173084 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.173302 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.173303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.173304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.173410 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.173663 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.173669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.173798 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.173898 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.174049 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260924 03:10:55.174674 14593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:55.174760 14593 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:55.174927 14593 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:55.175252 14133 ts_manager.cc:194] Registered new tserver with Master: b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
I20260924 03:10:55.175612 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53361
I20260924 03:10:55.182606 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:10:55.188881 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.183928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.0.98.62:40187)
0924 03:10:55.183964 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.183966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.184069 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:10:55.184114 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.184115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.184122 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.184164 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.184649 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.185049 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.185644 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.185793 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.185835 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.186820 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.186820 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.186821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.186825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.187930 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.188286 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.188287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.188287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.188332 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.188503 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.188504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.188504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.188622 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.188693 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.188776 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.188829 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:10:55.190448 14133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:10:55.192025 14133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:10:55.197961 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.0.1:37336)
0924 03:10:55.194525 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.194532 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.194588 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.195104 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.195105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.195107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.195157 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.195158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.196020 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.196021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.197041 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.197043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.197751 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.197752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.197802 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.197813 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.197882 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.197916 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":33,"threads_started":1}
I20260924 03:10:55.198019 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.0.98.3:33255)
0924 03:10:55.194913 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.194918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.194926 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:10:55.194958 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.194960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.194973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.195041 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.195234 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.195923 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.197119 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.197664 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.197746 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.197792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.197913 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.197948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":79,"threads_started":1}
I20260924 03:10:55.198566 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.0.1:55214)
0924 03:10:55.194424 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.194462 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.194544 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.194792 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.194795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.194797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.194870 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.194872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.196195 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.196196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.197405 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.197407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.198339 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.198340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.198406 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.198409 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.198467 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.198505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":31,"thread_start_us":76,"threads_started":1}
I20260924 03:10:55.198594 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.98.1:36513)
0924 03:10:55.194392 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.194397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.194646 (+ 249us) server_negotiation.cc:407] Connection header received
0924 03:10:55.194693 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.194695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.194707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.194763 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.194968 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.196080 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.197490 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.198254 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.198328 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.198383 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.198479 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.198517 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260924 03:10:55.198830 14527 tablet_service.cc:1467] Processing CreateTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 (DEFAULT_TABLE table=test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.199184 14527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55aafdbe3d04b11aeffc37e3747b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.199544 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45833 (local address 127.0.0.1:40628)
0924 03:10:55.194490 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.194500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.194553 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.194956 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.194957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.194959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.194996 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.194997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.196195 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.196196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.197405 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.197407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.199311 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.199311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.199376 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.199379 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.199442 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.199480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":61,"spinlock_wait_cycles":384,"thread_start_us":70,"threads_started":1}
I20260924 03:10:55.199467 14259 tablet_service.cc:1467] Processing CreateTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 (DEFAULT_TABLE table=test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.199579 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.194357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.98.2:45833)
0924 03:10:55.194695 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.194699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.194706 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:55.194736 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.194737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.194791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.194875 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.195070 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.196079 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.198417 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.199167 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.199227 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.199279 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.199465 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.199502 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":81,"threads_started":1}
I20260924 03:10:55.199762 14259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55aafdbe3d04b11aeffc37e3747b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.202749 14607 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Bootstrap starting.
I20260924 03:10:55.202760 14393 tablet_service.cc:1467] Processing CreateTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 (DEFAULT_TABLE table=test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.203053 14393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55aafdbe3d04b11aeffc37e3747b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.203094 14608 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Bootstrap starting.
I20260924 03:10:55.203460 14607 tablet_bootstrap.cc:654] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.203596 14608 tablet_bootstrap.cc:654] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.204032 14607 log.cc:824] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.204102 14608 log.cc:824] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.204818 14607 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: No bootstrap required, opened a new log
I20260924 03:10:55.204849 14608 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: No bootstrap required, opened a new log
I20260924 03:10:55.204879 14607 ts_tablet_manager.cc:1397] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:55.204896 14608 ts_tablet_manager.cc:1397] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:55.206254 14611 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Bootstrap starting.
I20260924 03:10:55.206709 14611 tablet_bootstrap.cc:654] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.206902 14607 raft_consensus.cc:348] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.206909 14608 raft_consensus.cc:348] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.207075 14607 raft_consensus.cc:374] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.207075 14608 raft_consensus.cc:374] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.207118 14608 raft_consensus.cc:729] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d305973b214c04b11464e1ca841e97, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.207118 14607 raft_consensus.cc:729] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68866397240470fad4d46f6e128a4d4, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.207195 14611 log.cc:824] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.207223 14607 consensus_queue.cc:260] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.207278 14608 consensus_queue.cc:260] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.207477 14607 ts_tablet_manager.cc:1428] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:55.207517 14608 ts_tablet_manager.cc:1428] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:10:55.207561 14325 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.207644 14593 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.208036 14611 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: No bootstrap required, opened a new log
I20260924 03:10:55.208101 14611 ts_tablet_manager.cc:1397] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:55.209558 14611 raft_consensus.cc:348] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.209743 14611 raft_consensus.cc:374] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.209774 14611 raft_consensus.cc:729] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1934512a46c5488db4ddffb2d3a1e3ae, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.209878 14611 consensus_queue.cc:260] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.210081 14611 ts_tablet_manager.cc:1428] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:55.210137 14459 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.246477 14613 raft_consensus.cc:482] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:55.246551 14613 raft_consensus.cc:504] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.246774 14613 leader_election.cc:288] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513)
I20260924 03:10:55.249647 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.246989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50235 (local address 127.0.98.2:45833)
0924 03:10:55.247024 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.247026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.247142 (+ 116us) server_negotiation.cc:407] Connection header received
0924 03:10:55.247199 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.247200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.247210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.247264 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.247416 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.247994 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.248945 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.249296 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.249374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.249383 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.249544 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.249593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:10:55.249653 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.246995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50065 (local address 127.0.98.1:36513)
0924 03:10:55.247036 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.247038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.247046 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:10:55.247069 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.247070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.247079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.247143 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.247318 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.247822 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.248910 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.249281 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.249370 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.249378 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.249548 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.249582 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:10:55.249652 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.246917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.98.3:50065)
0924 03:10:55.246951 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.246961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.246995 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.247185 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.247186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.247188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.247248 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.247250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.247899 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.247900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.248789 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.248792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.249372 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.249373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.249450 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.249452 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.249539 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.249577 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":32}
I20260924 03:10:55.249661 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.246917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45833 (local address 127.0.98.3:50235)
0924 03:10:55.247079 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.247089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.247139 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.247272 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.247272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.247273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.247353 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.247355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.248099 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.248100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.248821 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.248823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.249376 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.249376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.249445 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.249448 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.249535 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.249571 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":44,"thread_start_us":69,"threads_started":1}
I20260924 03:10:55.250015 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55aafdbe3d04b11aeffc37e3747b9f8" candidate_uuid: "b68866397240470fad4d46f6e128a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d305973b214c04b11464e1ca841e97" is_pre_election: true
I20260924 03:10:55.249990 14413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55aafdbe3d04b11aeffc37e3747b9f8" candidate_uuid: "b68866397240470fad4d46f6e128a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" is_pre_election: true
I20260924 03:10:55.250129 14413 raft_consensus.cc:2454] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68866397240470fad4d46f6e128a4d4 in term 0.
I20260924 03:10:55.250131 14279 raft_consensus.cc:2454] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68866397240470fad4d46f6e128a4d4 in term 0.
I20260924 03:10:55.250419 14482 leader_election.cc:302] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2d305973b214c04b11464e1ca841e97, b68866397240470fad4d46f6e128a4d4; no voters:
I20260924 03:10:55.250559 14613 raft_consensus.cc:2781] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:55.250612 14613 raft_consensus.cc:482] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:55.250633 14613 raft_consensus.cc:3037] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.251343 14613 raft_consensus.cc:504] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.251461 14613 leader_election.cc:288] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [CANDIDATE]: Term 1 election: Requested vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513)
I20260924 03:10:55.251617 14413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55aafdbe3d04b11aeffc37e3747b9f8" candidate_uuid: "b68866397240470fad4d46f6e128a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
I20260924 03:10:55.251704 14413 raft_consensus.cc:3037] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.252359 14413 raft_consensus.cc:2454] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68866397240470fad4d46f6e128a4d4 in term 1.
I20260924 03:10:55.252475 14279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55aafdbe3d04b11aeffc37e3747b9f8" candidate_uuid: "b68866397240470fad4d46f6e128a4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d305973b214c04b11464e1ca841e97"
I20260924 03:10:55.252543 14483 leader_election.cc:302] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4; no voters:
I20260924 03:10:55.252552 14279 raft_consensus.cc:3037] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.252637 14613 raft_consensus.cc:2781] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:55.252805 14613 raft_consensus.cc:686] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 LEADER]: Becoming Leader. State: Replica: b68866397240470fad4d46f6e128a4d4, State: Running, Role: LEADER
I20260924 03:10:55.252885 14613 consensus_queue.cc:237] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.253376 14279 raft_consensus.cc:2454] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68866397240470fad4d46f6e128a4d4 in term 1.
I20260924 03:10:55.253577 14134 catalog_manager.cc:5495] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b68866397240470fad4d46f6e128a4d4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b68866397240470fad4d46f6e128a4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } health_report { overall_health: UNKNOWN } } }
I20260924 03:10:55.269218 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.267208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.0.98.3:33255)
0924 03:10:55.267247 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.267249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.267256 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:10:55.267345 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.267346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.267360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.267427 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:55.267556 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.267943 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.268574 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.268740 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.268800 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.268939 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:55.269023 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:55.269092 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.269127 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.269143 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:10:55.270551 14279 raft_consensus.cc:1261] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Refusing update from remote peer b68866397240470fad4d46f6e128a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:55.270576 14413 raft_consensus.cc:1261] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Refusing update from remote peer b68866397240470fad4d46f6e128a4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:55.270831 14613 consensus_queue.cc:1035] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:55.270942 14617 consensus_queue.cc:1035] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:55.273252 14620 mvcc.cc:205] Tried to move back new op lower bound from 7332738888786280448 to 7332738888716394496. Current Snapshot: MvccSnapshot[applied={T|T < 7332738888786280448 or (T in {7332738888786280448})}]
I20260924 03:10:55.273365 14621 mvcc.cc:205] Tried to move back new op lower bound from 7332738888786280448 to 7332738888716394496. Current Snapshot: MvccSnapshot[applied={T|T < 7332738888786280448}]
I20260924 03:10:55.273890 14622 mvcc.cc:205] Tried to move back new op lower bound from 7332738888786280448 to 7332738888716394496. Current Snapshot: MvccSnapshot[applied={T|T < 7332738888786280448}]
I20260924 03:10:55.274921 14134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:55.274997 14134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:55.275660 14134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Sending DeleteTablet for 3 replicas of tablet e55aafdbe3d04b11aeffc37e3747b9f8
I20260924 03:10:55.275947 14393 tablet_service.cc:1514] Processing DeleteTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40628
I20260924 03:10:55.275967 14527 tablet_service.cc:1514] Processing DeleteTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37336
I20260924 03:10:55.276155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14196
I20260924 03:10:55.276155 14259 tablet_service.cc:1514] Processing DeleteTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55214
I20260924 03:10:55.277251 14631 tablet_replica.cc:330] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: stopping tablet replica
I20260924 03:10:55.277355 14631 raft_consensus.cc:2229] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:55.277452 14631 raft_consensus.cc:2258] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.277851 14631 ts_tablet_manager.cc:1905] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:55.279371 14631 ts_tablet_manager.cc:1918] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:55.279484 14631 log.cc:1196] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/e55aafdbe3d04b11aeffc37e3747b9f8
I20260924 03:10:55.279821 14631 ts_tablet_manager.cc:1939] T e55aafdbe3d04b11aeffc37e3747b9f8 P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting consensus metadata
I20260924 03:10:55.280270 14121 catalog_manager.cc:4841] TS 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): tablet e55aafdbe3d04b11aeffc37e3747b9f8 (table test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]) successfully deleted
W20260924 03:10:55.280938 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36513 (error 108)
I20260924 03:10:55.281368 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.281246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.0.1:55220)
0924 03:10:55.281287 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.281339 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:10:55.281559 14120 catalog_manager.cc:4584] TS a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e55aafdbe3d04b11aeffc37e3747b9f8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36513: connect: Connection refused (error 111)
I20260924 03:10:55.282063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14330
I20260924 03:10:55.282081 14630 tablet_replica.cc:330] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: stopping tablet replica
I20260924 03:10:55.286211 14630 raft_consensus.cc:2229] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:10:55.286392 14630 raft_consensus.cc:2258] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.286482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14464
I20260924 03:10:55.291316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14102
I20260924 03:10:55.291339 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.291246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.0.1:37360)
0924 03:10:55.291289 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.291314 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:10:55.295956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--webserver_interface=127.0.98.62
--webserver_port=44473
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:55.370077 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:55.370311 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:55.370335 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:55.371415 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:55.371452 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:55.371465 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:55.371500 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:55.371523 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:55.372905 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:55.373132 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:55.373355 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:55.380951 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.380964 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.380967 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.381283 14633 server_base.cc:1034] running on GCE node
I20260924 03:10:55.381573 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:55.381844 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.382984 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790219455382969 us; error 30 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.385174 14633 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:55.385609 14633 webserver.cc:460] Webserver started at http://127.0.98.62:44473/ using document root <none> and password file <none>
I20260924 03:10:55.385823 14633 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.385898 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.387131 14633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:55.387672 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.387818 14633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:55.387913 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "693631d2f02b49ef9682317ba512eee6"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "98c1ce7ff2e8d7465e62741885bb26c6"
server_key_iv: "29216c37b8bb7dd631db0d182f85d4df"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.388253 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.411841 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.413475 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.413609 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.417110 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40187
I20260924 03:10:55.417135 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40187 every 8 connection(s)
I20260924 03:10:55.417490 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:55.418855 14702 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:10:55.419327 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap starting.
I20260924 03:10:55.420241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14633
I20260924 03:10:55.420504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36513
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=46789
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:55.427435 14702 log.cc:824] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.429028 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:55.429270 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap complete.
I20260924 03:10:55.431448 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:55.431658 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.431825 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:55.431895 14702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:55.431951 14702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:55.431995 14702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:10:55.432893 14702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:55.433005 14702 leader_election.cc:302] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693631d2f02b49ef9682317ba512eee6; no voters:
I20260924 03:10:55.433167 14702 leader_election.cc:288] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:10:55.433218 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:10:55.433354 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 LEADER]: Becoming Leader. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Running, Role: LEADER
I20260924 03:10:55.433401 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:55.433518 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:55.433821 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:55.433923 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:55.433836 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693631d2f02b49ef9682317ba512eee6. Latest consensus state: current_term: 2 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:55.434222 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:55.434319 14712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:55.435146 14712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:55.435299 14712 catalog_manager.cc:1259] Loaded cluster ID: 2f2ae83ec4444415864ec77c47ac4aef
I20260924 03:10:55.435474 14712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:10:55.435690 14712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 03:10:55.438126 14712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 03:10:55.439801 14712 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:10:55.439983 14712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Loaded TSK: 0
I20260924 03:10:55.440176 14712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:10:55.508047 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:55.508206 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:55.508226 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:55.509249 14704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:55.509284 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:55.509307 14704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:55.510622 14704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46789
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:55.510798 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:55.510983 14704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:55.515448 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.515443 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.515522 14704 server_base.cc:1034] running on GCE node
W20260924 03:10:55.515606 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.515813 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:55.516043 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.517203 14704 hybrid_clock.cc:648] HybridClock initialized: now 1790219455517193 us; error 28 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.525835 14704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:55.526386 14704 webserver.cc:460] Webserver started at http://127.0.98.1:46789/ using document root <none> and password file <none>
I20260924 03:10:55.526612 14704 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.526684 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.528250 14704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:10:55.528964 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.529140 14704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:55.529230 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "a2d305973b214c04b11464e1ca841e97"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "5539e345f11b8e1c3c3886b57ba5cdfb"
server_key_iv: "2fc99e000251af7a212b239d8930996f"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.529608 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.554898 14704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.557204 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.557335 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.557550 14704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:55.557977 14746 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:10:55.559350 14704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:10:55.559406 14704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:10:55.559437 14704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:10:55.560267 14704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:10:55.560310 14704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:55.560385 14746 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Bootstrap starting.
I20260924 03:10:55.567018 14704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36513
I20260924 03:10:55.567461 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:55.570320 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36513 every 8 connection(s)
I20260924 03:10:55.570578 14746 log.cc:824] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.572944 14746 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:55.573232 14746 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Bootstrap complete.
I20260924 03:10:55.574150 14746 ts_tablet_manager.cc:1397] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20260924 03:10:55.575376 14746 raft_consensus.cc:348] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.575829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14704
I20260924 03:10:55.575822 14746 raft_consensus.cc:729] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d305973b214c04b11464e1ca841e97, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.575996 14746 consensus_queue.cc:260] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.576107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45833
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=40417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:55.576267 14746 ts_tablet_manager.cc:1428] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:55.580451 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.568143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:54403)
0924 03:10:55.568338 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.568384 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.568931 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.570763 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.570767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.571066 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.571316 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.571322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.572971 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.572972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.574065 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.574067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.574865 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.574866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.574996 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.575572 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.575581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.577144 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.578890 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.578897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.578899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.579100 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.579358 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.579358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.579359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.579458 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.579845 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.579852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.579986 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.580114 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.580270 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260924 03:10:55.580451 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.568266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54403 (local address 127.0.98.62:40187)
0924 03:10:55.570311 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.570320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.570354 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:10:55.570504 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.570511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.570564 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.571448 (+ 884us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.571528 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.572838 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.574155 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.574739 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.574855 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.577248 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.577252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.577252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.577264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.578781 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.579193 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.579196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.579196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.579282 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.579547 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.579548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.579548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.579749 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.579849 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.580175 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.580287 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":64,"threads_started":1}
I20260924 03:10:55.581154 14854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:55.581226 14854 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:55.581431 14854 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:55.582160 14666 ts_manager.cc:194] Registered new tserver with Master: a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513)
I20260924 03:10:55.583384 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54403
I20260924 03:10:55.584162 14854 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.589570 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.582831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.0.1:55222)
0924 03:10:55.582979 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.583008 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.583065 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.586440 (+ 3375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.586442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.586444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.586487 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.586488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.587696 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.587696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.588491 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.588493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.589358 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.589359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.589423 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.589433 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.589489 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.589523 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260924 03:10:55.589612 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.582868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.0.98.1:36513)
0924 03:10:55.586285 (+ 3417us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.586289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.586299 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:10:55.586329 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.586330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.586344 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.586429 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.586555 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.587602 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.588567 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.589270 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.589350 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.589396 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.589517 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.589551 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3388,"thread_start_us":74,"threads_started":1}
I20260924 03:10:55.589948 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55222
I20260924 03:10:55.590505 14863 tablet_replica.cc:330] stopping tablet replica
I20260924 03:10:55.590605 14863 raft_consensus.cc:2229] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:55.590669 14863 raft_consensus.cc:2258] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.590976 14863 ts_tablet_manager.cc:1905] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:55.593007 14863 ts_tablet_manager.cc:1918] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:55.593073 14863 log.cc:1196] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/e55aafdbe3d04b11aeffc37e3747b9f8
I20260924 03:10:55.593291 14863 ts_tablet_manager.cc:1939] T e55aafdbe3d04b11aeffc37e3747b9f8 P a2d305973b214c04b11464e1ca841e97: Deleting consensus metadata
I20260924 03:10:55.593755 14652 catalog_manager.cc:4841] TS a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513): tablet e55aafdbe3d04b11aeffc37e3747b9f8 (table test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]) successfully deleted
W20260924 03:10:55.654954 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:55.655134 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:55.655166 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:55.656241 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:55.656298 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:55.656337 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:55.657647 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40417
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:55.657881 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:55.658074 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:55.662529 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.662503 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.662663 14860 server_base.cc:1034] running on GCE node
W20260924 03:10:55.662504 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.662971 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:55.663151 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.664264 14860 hybrid_clock.cc:648] HybridClock initialized: now 1790219455664260 us; error 31 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.666445 14860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:55.666834 14860 webserver.cc:460] Webserver started at http://127.0.98.2:40417/ using document root <none> and password file <none>
I20260924 03:10:55.667037 14860 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.667085 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.668275 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:55.668815 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.669009 14860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.669088 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "8617d4955b2d260a0f379ee596db80f9"
server_key_iv: "1268c2150ebd176c9ce262bbf0918b6e"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.669363 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.695632 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.697060 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.697190 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.697365 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:55.697606 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:55.697635 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.697652 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:55.697691 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.703152 14860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45833
I20260924 03:10:55.703199 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45833 every 8 connection(s)
I20260924 03:10:55.703490 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:55.711020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14860
I20260924 03:10:55.711285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33255
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=40503
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:55.711442 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.704153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47171 (local address 127.0.98.62:40187)
0924 03:10:55.704208 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.704210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.704399 (+ 189us) server_negotiation.cc:407] Connection header received
0924 03:10:55.704805 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.704806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.704819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.704901 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.705692 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.706126 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.706981 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.707151 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.707196 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.708896 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.708897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.708897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.708903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.710227 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.710561 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.710562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.710562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.710609 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.710801 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.710802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.710802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.710932 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.711010 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.711235 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.711317 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:10:55.711553 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.704118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:47171)
0924 03:10:55.704274 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.704309 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.704713 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.704952 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.704955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.705452 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.705620 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.705623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.706239 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.706240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.706915 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.706917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.707214 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.707214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.707294 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.707652 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.707658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.708822 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.710290 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.710295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.710296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.710501 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.710663 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.710663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.710663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.710754 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.711002 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.711006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.711125 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.711206 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.711305 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260924 03:10:55.712216 14992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:55.712294 14992 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:55.712466 14992 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:55.712806 14666 ts_manager.cc:194] Registered new tserver with Master: 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833)
I20260924 03:10:55.713168 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47171
W20260924 03:10:55.787180 14995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:55.787369 14995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:55.787401 14995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:55.788471 14995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:55.788534 14995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:55.788578 14995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:55.789966 14995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40503
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:55.790225 14995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:55.790441 14995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:55.794987 15003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.795017 15001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:55.795017 15000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:55.795356 14995 server_base.cc:1034] running on GCE node
I20260924 03:10:55.795560 14995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:55.795811 14995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:55.796969 14995 hybrid_clock.cc:648] HybridClock initialized: now 1790219455796958 us; error 24 us; skew 500 ppm
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:55.799297 14995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:55.799714 14995 webserver.cc:460] Webserver started at http://127.0.98.3:40503/ using document root <none> and password file <none>
I20260924 03:10:55.799928 14995 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:55.800012 14995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:55.801149 14995 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:55.801731 15010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.801889 14995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:55.802016 14995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "b68866397240470fad4d46f6e128a4d4"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "632331cafd3d781719c8bc075319f807"
server_key_iv: "3df4616f40306a5cd8edb80d3218011d"
server_key_version: "encryptionkey@0"
I20260924 03:10:55.802366 14995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:55.817874 14995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:55.819914 14995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:55.820076 14995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:55.820272 14995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:55.820643 15017 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:10:55.821892 14995 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:10:55.821954 14995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:55.822012 14995 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:10:55.822631 14995 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:10:55.822685 14995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:55.822729 15017 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Bootstrap starting.
I20260924 03:10:55.829241 14995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260924 03:10:55.829339 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260924 03:10:55.829638 14995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:55.831068 15017 log.cc:824] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.833040 15017 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:55.833372 15017 tablet_bootstrap.cc:492] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Bootstrap complete.
I20260924 03:10:55.834488 15017 ts_tablet_manager.cc:1397] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:10:55.835469 15017 raft_consensus.cc:348] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.835742 15017 raft_consensus.cc:729] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68866397240470fad4d46f6e128a4d4, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.835886 15017 consensus_queue.cc:260] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } }
I20260924 03:10:55.836128 15017 ts_tablet_manager.cc:1428] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:10:55.836447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 14995
I20260924 03:10:55.838529 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.830302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52499 (local address 127.0.98.62:40187)
0924 03:10:55.830341 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.830343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.830572 (+ 229us) server_negotiation.cc:407] Connection header received
0924 03:10:55.831014 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.831016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.831027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.831082 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.831673 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.832181 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.833318 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.833579 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.833655 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.835841 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.835842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.835843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.835851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.837239 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.837602 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.837603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.837604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.837668 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.837903 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.837904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.837904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.838010 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.838079 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.838378 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.838456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:10:55.838680 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.830284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:52499)
0924 03:10:55.830462 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.830493 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.830910 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.831130 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.831134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.831379 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.831580 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.831584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.832292 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.832293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.833182 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.833185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.833678 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.833679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.833802 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.834318 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.834325 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.835751 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.837327 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.837333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.837334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.837531 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.837731 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.837732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.837732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.837845 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.838081 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.838087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.838257 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.838358 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.838484 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260924 03:10:55.839058 15125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260924 03:10:55.839155 15125 heartbeater.cc:464] Registering TS with master...
I20260924 03:10:55.839320 15125 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260924 03:10:55.839854 14666 ts_manager.cc:194] Registered new tserver with Master: b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
I20260924 03:10:55.840420 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52499
I20260924 03:10:55.841074 15125 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.843164 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.836730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.0.98.62:40187)
0924 03:10:55.836864 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.836866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.836875 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:10:55.836942 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.836943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.836954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.837024 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.837524 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.837944 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.838687 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.838987 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.839082 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.840387 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.840388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.840389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.840395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.842074 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.842441 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.842442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.842442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.842503 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.842687 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.842688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.842688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.842794 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.842959 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.842995 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.843068 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":55,"threads_started":1}
I20260924 03:10:55.843873 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:10:55 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 03:10:55.846266 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.844188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.98.62:40187)
0924 03:10:55.844224 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.844226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.844288 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:10:55.844368 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.844368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.844397 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.844474 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:55.844558 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.844992 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.845648 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.845806 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.845850 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.846003 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:55.846071 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:55.846154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.846174 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.846209 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:10:55.847960 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.840132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.0.1:37366)
0924 03:10:55.840175 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.840184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.840215 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.840909 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.840910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.841263 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:55.841324 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.841326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.842502 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.842503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.842943 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.842945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.843596 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.843600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.843651 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.844087 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:55.844092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:55.845303 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:55.847102 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.847107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.847108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.847267 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.847463 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:55.847464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:55.847464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:55.847509 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:55.847761 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:55.847766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:55.847800 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.847844 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.847905 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:10:55.848024 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.840284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.98.3:33255)
0924 03:10:55.840605 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.840609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.840619 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:10:55.840648 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.840650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.840662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.840784 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:55.841429 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.842426 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.843038 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.843500 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.843581 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.845383 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:55.845390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:55.845390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:55.845396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:55.847019 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.847337 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.847339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.847340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.847395 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:55.847563 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:55.847564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:55.847564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:55.847702 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:55.847774 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.847872 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.847952 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":66,"threads_started":1}
I20260924 03:10:55.846486 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:10:55.848368 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet e55aafdbe3d04b11aeffc37e3747b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
W20260924 03:10:55.848644 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:10:55.848877 15134 tablet_replica.cc:330] stopping tablet replica
I20260924 03:10:55.848981 15134 raft_consensus.cc:2229] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:55.849045 15134 raft_consensus.cc:2258] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.849328 15134 ts_tablet_manager.cc:1905] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:55.850519 15059 tablet_service.cc:1467] Processing CreateTablet for tablet bcdcaebb07d645c1b85499c8635b99ef (DEFAULT_TABLE table=test-table [id=23ed1d59e9ff46e3a8847ecb897a0247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.850526 14788 tablet_service.cc:1467] Processing CreateTablet for tablet bcdcaebb07d645c1b85499c8635b99ef (DEFAULT_TABLE table=test-table [id=23ed1d59e9ff46e3a8847ecb897a0247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.850651 15059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdcaebb07d645c1b85499c8635b99ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.850689 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdcaebb07d645c1b85499c8635b99ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.851604 15017 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Bootstrap starting.
I20260924 03:10:55.851824 14746 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Bootstrap starting.
I20260924 03:10:55.851864 15017 tablet_bootstrap.cc:654] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.852089 14746 tablet_bootstrap.cc:654] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.852725 14746 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: No bootstrap required, opened a new log
I20260924 03:10:55.852774 14746 ts_tablet_manager.cc:1397] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:55.852891 14746 raft_consensus.cc:348] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.852967 14746 raft_consensus.cc:374] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.852979 15134 ts_tablet_manager.cc:1918] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:55.853008 14746 raft_consensus.cc:729] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d305973b214c04b11464e1ca841e97, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.853042 15134 log.cc:1196] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/e55aafdbe3d04b11aeffc37e3747b9f8
I20260924 03:10:55.853051 14746 consensus_queue.cc:260] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.853129 14746 ts_tablet_manager.cc:1428] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.853276 15134 ts_tablet_manager.cc:1939] T e55aafdbe3d04b11aeffc37e3747b9f8 P b68866397240470fad4d46f6e128a4d4: Deleting consensus metadata
I20260924 03:10:55.853662 14653 catalog_manager.cc:4841] TS b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): tablet e55aafdbe3d04b11aeffc37e3747b9f8 (table test-table [id=5a0d1075ec9349fb8ea6721d76428b9e]) successfully deleted
I20260924 03:10:55.853948 15017 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: No bootstrap required, opened a new log
I20260924 03:10:55.854000 15017 ts_tablet_manager.cc:1397] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:10:55.854127 15017 raft_consensus.cc:348] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.854230 15017 raft_consensus.cc:374] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.854269 15017 raft_consensus.cc:729] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68866397240470fad4d46f6e128a4d4, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.854328 15017 consensus_queue.cc:260] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.854411 15017 ts_tablet_manager.cc:1428] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:55.855508 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.850297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45833 (local address 127.0.0.1:40632)
0924 03:10:55.851804 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.851813 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.851853 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.852063 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.852064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.852065 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.852116 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.852118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.853486 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.853487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.854633 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.854635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.855347 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.855348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.855390 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.855392 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.855436 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.855470 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494}
I20260924 03:10:55.855521 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.850387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.98.2:45833)
0924 03:10:55.850556 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.850560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.851908 (+ 1348us) server_negotiation.cc:407] Connection header received
0924 03:10:55.851950 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.851953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.851969 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.852214 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.852286 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.853377 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.854722 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.855289 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.855341 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.855373 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.855445 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.855468 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260924 03:10:55.856123 14926 tablet_service.cc:1467] Processing CreateTablet for tablet bcdcaebb07d645c1b85499c8635b99ef (DEFAULT_TABLE table=test-table [id=23ed1d59e9ff46e3a8847ecb897a0247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:10:55.856415 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdcaebb07d645c1b85499c8635b99ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:10:55.858826 15138 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Bootstrap starting.
I20260924 03:10:55.859380 15138 tablet_bootstrap.cc:654] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Neither blocks nor log segments found. Creating new log.
I20260924 03:10:55.859793 15138 log.cc:824] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:55.860337 15138 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: No bootstrap required, opened a new log
I20260924 03:10:55.860409 15138 ts_tablet_manager.cc:1397] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:10:55.861605 15138 raft_consensus.cc:348] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.861722 15138 raft_consensus.cc:374] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:10:55.861742 15138 raft_consensus.cc:729] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1934512a46c5488db4ddffb2d3a1e3ae, State: Initialized, Role: FOLLOWER
I20260924 03:10:55.861841 15138 consensus_queue.cc:260] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.862123 15138 ts_tablet_manager.cc:1428] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:10:55.862140 14992 heartbeater.cc:503] Master 127.0.98.62:40187 was elected leader, sending a full tablet report...
I20260924 03:10:55.936273 15140 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:55.936362 15140 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.936650 15140 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
I20260924 03:10:55.939368 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.936841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52849 (local address 127.0.98.1:36513)
0924 03:10:55.936886 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.936888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.936896 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:10:55.936929 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.936930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.936942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.936986 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.937124 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.937620 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.938643 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.939032 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.939083 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.939092 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.939256 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.939292 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:10:55.939352 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.936775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.98.2:52849)
0924 03:10:55.936825 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.936836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.936873 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.936988 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.936989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.936990 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.937056 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.937058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.937696 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.937696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.938564 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.938566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.939111 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.939112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.939187 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.939192 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.939243 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.939279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":49}
I20260924 03:10:55.939522 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.936806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.98.2:49839)
0924 03:10:55.936980 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.936988 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:55.937035 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:55.937173 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:55.937174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:55.937176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:10:55.937235 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.937236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.937856 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.937857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.938662 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:55.938665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.939251 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.939252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:55.939323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.939327 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:10:55.939383 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:10:55.939422 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260924 03:10:55.939533 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.936900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49839 (local address 127.0.98.3:33255)
0924 03:10:55.936941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.936944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.937029 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:10:55.937080 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.937081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.937100 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.937156 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:10:55.937297 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.937777 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.938742 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.939154 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.939226 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.939288 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.939405 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.939450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:10:55.939785 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdcaebb07d645c1b85499c8635b99ef" candidate_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68866397240470fad4d46f6e128a4d4" is_pre_election: true
I20260924 03:10:55.939929 15079 raft_consensus.cc:2454] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1934512a46c5488db4ddffb2d3a1e3ae in term 0.
I20260924 03:10:55.940078 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdcaebb07d645c1b85499c8635b99ef" candidate_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d305973b214c04b11464e1ca841e97" is_pre_election: true
I20260924 03:10:55.940153 14882 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4; no voters:
I20260924 03:10:55.940217 14808 raft_consensus.cc:2454] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1934512a46c5488db4ddffb2d3a1e3ae in term 0.
I20260924 03:10:55.940294 15140 raft_consensus.cc:2781] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:10:55.940387 15140 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:10:55.940433 15140 raft_consensus.cc:3037] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.941073 15140 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.941196 15140 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [CANDIDATE]: Term 1 election: Requested vote from peers a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
I20260924 03:10:55.941339 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdcaebb07d645c1b85499c8635b99ef" candidate_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d305973b214c04b11464e1ca841e97"
I20260924 03:10:55.941375 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdcaebb07d645c1b85499c8635b99ef" candidate_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68866397240470fad4d46f6e128a4d4"
I20260924 03:10:55.941448 15079 raft_consensus.cc:3037] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.941442 14808 raft_consensus.cc:3037] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:10:55.942250 15079 raft_consensus.cc:2454] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1934512a46c5488db4ddffb2d3a1e3ae in term 1.
I20260924 03:10:55.942278 14808 raft_consensus.cc:2454] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1934512a46c5488db4ddffb2d3a1e3ae in term 1.
I20260924 03:10:55.942446 14881 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1934512a46c5488db4ddffb2d3a1e3ae, a2d305973b214c04b11464e1ca841e97; no voters:
I20260924 03:10:55.942548 15140 raft_consensus.cc:2781] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:10:55.942672 15140 raft_consensus.cc:686] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 LEADER]: Becoming Leader. State: Replica: 1934512a46c5488db4ddffb2d3a1e3ae, State: Running, Role: LEADER
I20260924 03:10:55.942776 15140 consensus_queue.cc:237] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:55.943476 14666 catalog_manager.cc:5495] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } health_report { overall_health: UNKNOWN } } }
W20260924 03:10:55.954005 14993 tablet.cc:2369] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:10:55.968578 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.966751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.0.98.2:45833)
0924 03:10:55.966788 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:55.966789 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:55.966795 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:10:55.966856 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:55.966857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:55.966870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:55.966922 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:10:55.967007 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.967380 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.968028 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:55.968189 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:55.968234 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:55.968352 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:10:55.968436 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:10:55.968490 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:55.968522 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:10:55.968537 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:10:55.970070 15079 raft_consensus.cc:1261] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Refusing update from remote peer 1934512a46c5488db4ddffb2d3a1e3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:55.970088 14808 raft_consensus.cc:1261] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Refusing update from remote peer 1934512a46c5488db4ddffb2d3a1e3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:10:55.970357 15140 consensus_queue.cc:1035] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:55.970487 15142 consensus_queue.cc:1035] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:10:55.972150 15145 mvcc.cc:205] Tried to move back new op lower bound from 7332738891650977792 to 7332738891542208512. Current Snapshot: MvccSnapshot[applied={T|T < 7332738891650977792}]
I20260924 03:10:55.972821 15146 mvcc.cc:205] Tried to move back new op lower bound from 7332738891650977792 to 7332738891542208512. Current Snapshot: MvccSnapshot[applied={T|T < 7332738891650977792 or (T in {7332738891650977792})}]
I20260924 03:10:55.973310 15147 mvcc.cc:205] Tried to move back new op lower bound from 7332738891650977792 to 7332738891542208512. Current Snapshot: MvccSnapshot[applied={T|T < 7332738891650977792 or (T in {7332738891650977792})}]
I20260924 03:10:55.974783 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:55.974870 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:10:55.975526 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Sending DeleteTablet for 3 replicas of tablet bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:10:55.975777 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet bcdcaebb07d645c1b85499c8635b99ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55222
I20260924 03:10:55.975796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14704
I20260924 03:10:55.975823 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet bcdcaebb07d645c1b85499c8635b99ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260924 03:10:55.975908 14863 tablet_replica.cc:330] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: stopping tablet replica
I20260924 03:10:55.975951 14863 raft_consensus.cc:2229] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:55.975994 14863 raft_consensus.cc:2258] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.976321 15134 tablet_replica.cc:330] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: stopping tablet replica
I20260924 03:10:55.976368 15134 raft_consensus.cc:2229] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:10:55.976408 15134 raft_consensus.cc:2258] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.976720 14926 tablet_service.cc:1514] Processing DeleteTablet for tablet bcdcaebb07d645c1b85499c8635b99ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40632
I20260924 03:10:55.976924 15154 tablet_replica.cc:330] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: stopping tablet replica
I20260924 03:10:55.977015 15154 raft_consensus.cc:2229] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:10:55.977034 15134 ts_tablet_manager.cc:1905] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:55.977116 15154 raft_consensus.cc:2258] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:10:55.978497 15134 ts_tablet_manager.cc:1918] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:55.978508 15154 ts_tablet_manager.cc:1905] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:10:55.978569 15134 log.cc:1196] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:10:55.978771 15134 ts_tablet_manager.cc:1939] T bcdcaebb07d645c1b85499c8635b99ef P b68866397240470fad4d46f6e128a4d4: Deleting consensus metadata
I20260924 03:10:55.979058 14653 catalog_manager.cc:4841] TS b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): tablet bcdcaebb07d645c1b85499c8635b99ef (table test-table [id=23ed1d59e9ff46e3a8847ecb897a0247]) successfully deleted
I20260924 03:10:55.980027 15154 ts_tablet_manager.cc:1918] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:10:55.980113 15154 log.cc:1196] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:10:55.980350 15154 ts_tablet_manager.cc:1939] T bcdcaebb07d645c1b85499c8635b99ef P 1934512a46c5488db4ddffb2d3a1e3ae: Deleting consensus metadata
I20260924 03:10:55.980691 14653 catalog_manager.cc:4841] TS 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): tablet bcdcaebb07d645c1b85499c8635b99ef (table test-table [id=23ed1d59e9ff46e3a8847ecb897a0247]) successfully deleted
W20260924 03:10:55.980748 14652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36513 (error 108)
I20260924 03:10:55.981128 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:55.981021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36513 (local address 127.0.0.1:55236)
0924 03:10:55.981069 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:55.981099 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:10:55.981227 14652 catalog_manager.cc:4584] TS a2d305973b214c04b11464e1ca841e97 (127.0.98.1:36513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcdcaebb07d645c1b85499c8635b99ef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36513: connect: Connection refused (error 111)
I20260924 03:10:55.982033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14860
I20260924 03:10:55.987237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14995
I20260924 03:10:55.991544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 14633
I20260924 03:10:55.996124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--webserver_interface=127.0.98.62
--webserver_port=44473
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:56.070333 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:56.070509 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:56.070530 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:56.071524 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:10:56.071553 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:56.071566 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:56.071578 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:10:56.071589 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:10:56.072909 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:56.073073 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:56.073215 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:56.077473 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.077500 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.077448 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:56.077674 15156 server_base.cc:1034] running on GCE node
I20260924 03:10:56.077893 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:56.078161 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:56.079337 15156 hybrid_clock.cc:648] HybridClock initialized: now 1790219456079327 us; error 27 us; skew 500 ppm
Sep 24 03:10:56 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:56.081485 15156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:10:56.081863 15156 webserver.cc:460] Webserver started at http://127.0.98.62:44473/ using document root <none> and password file <none>
I20260924 03:10:56.082075 15156 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:56.082137 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:56.083417 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:10:56.084026 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.084194 15156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:10:56.084280 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "693631d2f02b49ef9682317ba512eee6"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "98c1ce7ff2e8d7465e62741885bb26c6"
server_key_iv: "29216c37b8bb7dd631db0d182f85d4df"
server_key_version: "encryptionkey@0"
I20260924 03:10:56.084589 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:56.092283 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:56.093941 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:56.094080 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:56.097610 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40187
I20260924 03:10:56.097637 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40187 every 8 connection(s)
I20260924 03:10:56.097990 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:10:56.099337 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:10:56.099852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15156
I20260924 03:10:56.099983 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap starting.
I20260924 03:10:56.100126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36513
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=46789
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:56.108349 15224 log.cc:824] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:56.110410 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:56.110714 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6: Bootstrap complete.
I20260924 03:10:56.112800 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:56.112994 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Initialized, Role: FOLLOWER
I20260924 03:10:56.113149 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:56.113204 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:10:56.113250 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:10:56.113296 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:10:56.114289 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:56.114367 15224 leader_election.cc:302] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693631d2f02b49ef9682317ba512eee6; no voters:
I20260924 03:10:56.114535 15224 leader_election.cc:288] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:10:56.114584 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:10:56.114753 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [term 3 LEADER]: Becoming Leader. State: Replica: 693631d2f02b49ef9682317ba512eee6, State: Running, Role: LEADER
I20260924 03:10:56.114776 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:10:56.114880 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260924 03:10:56.115284 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:56.115355 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:56.115851 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693631d2f02b49ef9682317ba512eee6. Latest consensus state: current_term: 3 leader_uuid: "693631d2f02b49ef9682317ba512eee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693631d2f02b49ef9682317ba512eee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260924 03:10:56.116003 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693631d2f02b49ef9682317ba512eee6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:10:56.116166 15234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:10:56.116933 15234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:10:56.117048 15234 catalog_manager.cc:1259] Loaded cluster ID: 2f2ae83ec4444415864ec77c47ac4aef
I20260924 03:10:56.117082 15234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 03:10:56.117285 15234 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff69d2f19ad google::LogMessage::Fail() at ??:0
@ 0x7ff69d2f488a google::LogMessage::SendToLog() at ??:0
@ 0x7ff69d2f14de google::LogMessage::Flush() at ??:0
@ 0x7ff69d2f2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff6a52120f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff6a521b8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff69f1bf0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff69f1b4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6a00186db start_thread at ??:0
@ 0x7ff69bdef61f clone at ??:0
W20260924 03:10:56.187304 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:56.187511 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:56.187549 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:56.188658 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:56.188720 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:56.188764 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:10:56.190276 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46789
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:56.190501 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:56.190712 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:56.195191 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.195180 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:56.195398 15226 server_base.cc:1034] running on GCE node
W20260924 03:10:56.195168 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:56.195600 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:56.195822 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:56.196964 15226 hybrid_clock.cc:648] HybridClock initialized: now 1790219456196950 us; error 33 us; skew 500 ppm
Sep 24 03:10:56 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:56.199287 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:10:56.199710 15226 webserver.cc:460] Webserver started at http://127.0.98.1:46789/ using document root <none> and password file <none>
I20260924 03:10:56.199913 15226 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:56.199980 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:56.201215 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:56.201951 15271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.202158 15226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:56.202271 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "a2d305973b214c04b11464e1ca841e97"
format_stamp: "Formatted at 2026-09-24 03:10:54 on dist-test-slave-jtnz"
server_key: "5539e345f11b8e1c3c3886b57ba5cdfb"
server_key_iv: "2fc99e000251af7a212b239d8930996f"
server_key_version: "encryptionkey@0"
I20260924 03:10:56.202585 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:56.226877 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:56.228976 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:56.229136 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:56.229362 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:56.229784 15278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:10:56.231765 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:10:56.231820 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:10:56.231855 15226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:10:56.232470 15226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:10:56.232524 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:10:56.232599 15278 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Bootstrap starting.
I20260924 03:10:56.239351 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36513
I20260924 03:10:56.239391 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36513 every 8 connection(s)
I20260924 03:10:56.239771 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:10:56.243301 15278 log.cc:824] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:10:56.245090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15226
I20260924 03:10:56.245333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45833
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=40417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:10:56.245685 15278 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:10:56.245981 15278 tablet_bootstrap.cc:492] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Bootstrap complete.
I20260924 03:10:56.247154 15278 ts_tablet_manager.cc:1397] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260924 03:10:56.248667 15278 raft_consensus.cc:348] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:56.249086 15278 raft_consensus.cc:729] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d305973b214c04b11464e1ca841e97, State: Initialized, Role: FOLLOWER
I20260924 03:10:56.249284 15278 consensus_queue.cc:260] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:56.249584 15278 ts_tablet_manager.cc:1428] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260924 03:10:56.311168 15388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:10:56.240328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:57283)
0924 03:10:56.240495 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.240535 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:56.240942 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:56.310980 (+ 70038us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260924 03:10:56.311424 15386 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40187: Client connection negotiation failed: client connection to 127.0.98.62:40187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:10:56.311744 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.311622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:35601)
0924 03:10:56.311676 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.311718 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:10:56.312021 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.311934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:37827)
0924 03:10:56.311976 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.312000 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:10:56.312130 15386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:10:56.320549 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:56.320693 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:56.320711 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:56.321740 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:56.321772 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:56.321796 15390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:10:56.323175 15390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40417
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:56.323346 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:56.323506 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:56.327739 15399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.327731 15397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.327732 15396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:56.327792 15390 server_base.cc:1034] running on GCE node
I20260924 03:10:56.328095 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:56.328333 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:56.329499 15390 hybrid_clock.cc:648] HybridClock initialized: now 1790219456329492 us; error 51 us; skew 500 ppm
Sep 24 03:10:56 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:56.331809 15390 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:10:56.332260 15390 webserver.cc:460] Webserver started at http://127.0.98.2:40417/ using document root <none> and password file <none>
I20260924 03:10:56.332476 15390 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:56.332541 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:56.333721 15390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:56.334262 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.334422 15390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.334517 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "1934512a46c5488db4ddffb2d3a1e3ae"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "8617d4955b2d260a0f379ee596db80f9"
server_key_iv: "1268c2150ebd176c9ce262bbf0918b6e"
server_key_version: "encryptionkey@0"
I20260924 03:10:56.334806 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:56.346087 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:56.347620 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:56.347754 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:56.347965 15390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:56.348232 15390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:56.348279 15390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.348315 15390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:56.348340 15390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.353574 15390 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45833
I20260924 03:10:56.353617 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45833 every 8 connection(s)
I20260924 03:10:56.353904 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:10:56.354732 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.354419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:42115)
0924 03:10:56.354566 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.354620 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
W20260924 03:10:56.355050 15520 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40187: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
I20260924 03:10:56.355266 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.355190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:38993)
0924 03:10:56.355229 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.355251 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:10:56.355464 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.355420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:59935)
0924 03:10:56.355438 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.355454 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260924 03:10:56.355546 15520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:10:56.359732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15390
I20260924 03:10:56.359970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33255
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40187
--webserver_port=40503
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:10:56.433849 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:10:56.434031 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:10:56.434067 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:10:56.435204 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:10:56.435266 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:10:56.435308 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:10:56.436678 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40503
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:10:56.436901 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:10:56.437106 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:10:56.441648 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.441634 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:10:56.441637 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:10:56.441902 15523 server_base.cc:1034] running on GCE node
I20260924 03:10:56.442109 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:10:56.442373 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:10:56.443534 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790219456443527 us; error 37 us; skew 500 ppm
Sep 24 03:10:56 dist-test-slave-jtnz krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:10:56.445741 15523 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:10:56.446213 15523 webserver.cc:460] Webserver started at http://127.0.98.3:40503/ using document root <none> and password file <none>
I20260924 03:10:56.446439 15523 fs_manager.cc:359] Metadata directory not provided
I20260924 03:10:56.446507 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:10:56.447587 15523 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:10:56.448103 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.448264 15523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:10:56.448346 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "b68866397240470fad4d46f6e128a4d4"
format_stamp: "Formatted at 2026-09-24 03:10:55 on dist-test-slave-jtnz"
server_key: "632331cafd3d781719c8bc075319f807"
server_key_iv: "3df4616f40306a5cd8edb80d3218011d"
server_key_version: "encryptionkey@0"
I20260924 03:10:56.448623 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:10:56.465647 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:10:56.467378 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:10:56.467526 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:10:56.467733 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:10:56.468004 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:10:56.468055 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.468091 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:10:56.468117 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:10:56.473767 15523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260924 03:10:56.473811 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260924 03:10:56.474124 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:10:56.475083 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.474749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:35517)
0924 03:10:56.474891 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.474959 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260924 03:10:56.475463 15652 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40187: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
I20260924 03:10:56.475706 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.475625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:33605)
0924 03:10:56.475666 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.475691 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:10:56.475944 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:56.475885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:35349)
0924 03:10:56.475907 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:56.475925 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 03:10:56.476055 15652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:10:56.484161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15523
I20260924 03:10:57.312674 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:57.312458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:51663)
0924 03:10:57.312586 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:57.312618 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 03:10:57.356025 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:57.355838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:56149)
0924 03:10:57.355961 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:57.355985 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260924 03:10:57.476540 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:57.476341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:58959)
0924 03:10:57.476468 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:57.476498 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 03:10:58.062446 15659 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:10:58.062557 15659 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:10:58.062817 15659 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
Sep 24 03:10:58 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 03:10:58 dist-test-slave-jtnz krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 03:10:58.071967 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.063146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.98.1:37109)
0924 03:10:58.063236 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:58.063245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:58.063279 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:58.064009 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:58.064010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:58.064356 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:58.064523 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:58.064528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.065515 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.065516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:58.066241 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:58.066243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.066766 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:58.066864 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:58.067448 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:58.067453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:58.068964 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:58.070614 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:58.070615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:58.070615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:58.070822 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:58.071042 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:58.071043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:58.071043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:58.071145 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:58.071494 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:58.071499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:58.071603 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 03:10:58.071709 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:10:58.071877 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260924 03:10:58.071978 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.063092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51059 (local address 127.0.98.2:45833)
0924 03:10:58.063269 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:58.063284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:58.063310 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:10:58.063774 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:58.063777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:58.063815 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:58.063936 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:58.064630 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.065394 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066341 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.066672 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066791 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:58.068728 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:58.068736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:58.068736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:58.068745 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:58.070466 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:58.070923 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:58.070926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:58.070926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:58.071022 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:58.071232 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:58.071232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:58.071233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:58.071419 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:58.071500 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:58.071753 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:10:58.071867 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260924 03:10:58.071997 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.063260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37109 (local address 127.0.98.3:33255)
0924 03:10:58.063404 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:10:58.063412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:10:58.063436 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:10:58.063829 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:10:58.063830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:10:58.063862 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:10:58.064001 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:10:58.064631 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.065398 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066307 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.066673 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066781 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:58.069069 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:10:58.069072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:10:58.069072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:10:58.069080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:10:58.070541 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:58.070891 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:58.070894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:58.070895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:58.070981 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:10:58.071205 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:10:58.071206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:10:58.071207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:10:58.071412 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:10:58.071498 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:10:58.071759 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:10:58.071894 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260924 03:10:58.072398 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.062997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45833 (local address 127.0.98.1:51059)
0924 03:10:58.063104 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:58.063115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:10:58.063171 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:10:58.064000 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:10:58.064002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:10:58.064357 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:10:58.064523 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:58.064528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.065489 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.065490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:58.066241 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:10:58.066243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:10:58.066777 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:10:58.066778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:10:58.066864 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:10:58.067386 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:10:58.067393 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:10:58.068629 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:10:58.070537 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:58.070543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:58.070545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:58.070824 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:58.071095 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:10:58.071096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:10:58.071096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:10:58.071155 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:10:58.071496 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:10:58.071502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:10:58.071604 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 03:10:58.071709 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:10:58.071959 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":180,"thread_start_us":44,"threads_started":1}
W20260924 03:10:58.073068 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
W20260924 03:10:58.073163 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:10:58.073189 15275 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2d305973b214c04b11464e1ca841e97; no voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4
I20260924 03:10:58.073274 15659 raft_consensus.cc:2726] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:10:58.313194 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.313061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:57399)
0924 03:10:58.313128 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:58.313154 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 03:10:58.356598 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.356378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:37115)
0924 03:10:58.356509 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:58.356560 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260924 03:10:58.477223 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:58.477034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:48997)
0924 03:10:58.477148 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:58.477186 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260924 03:10:59.313798 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:59.313593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:41309)
0924 03:10:59.313726 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:59.313754 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260924 03:10:59.357118 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:59.356934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:55591)
0924 03:10:59.357046 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:59.357081 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":47,"threads_started":1}
I20260924 03:10:59.477778 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:10:59.477593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:35709)
0924 03:10:59.477719 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:10:59.477745 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260924 03:11:00.003731 15669 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:00.003813 15669 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:11:00.003981 15669 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
W20260924 03:11:00.004390 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
W20260924 03:11:00.004454 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:11:00.004469 15275 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2d305973b214c04b11464e1ca841e97; no voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4
I20260924 03:11:00.004531 15669 raft_consensus.cc:2726] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:00.314435 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:00.314224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:47921)
0924 03:11:00.314347 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:00.314377 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260924 03:11:00.357781 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:00.357607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:58361)
0924 03:11:00.357713 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:00.357741 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260924 03:11:00.478389 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:00.478147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:33581)
0924 03:11:00.478307 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:00.478336 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260924 03:11:01.315078 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:01.314888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:54449)
0924 03:11:01.315002 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:01.315032 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260924 03:11:01.358494 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:01.358301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:58543)
0924 03:11:01.358432 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:01.358462 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 03:11:01.478931 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:01.478756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:40625)
0924 03:11:01.478864 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:01.478898 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260924 03:11:02.315853 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:02.315574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:37779)
0924 03:11:02.315774 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:02.315807 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
I20260924 03:11:02.359150 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:02.358982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:56621)
0924 03:11:02.359084 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:02.359116 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 03:11:02.479570 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:02.479334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:42403)
0924 03:11:02.479493 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:02.479527 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260924 03:11:02.691769 15679 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:02.691885 15679 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:11:02.692078 15679 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
W20260924 03:11:02.692500 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
W20260924 03:11:02.692595 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:11:02.692624 15275 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2d305973b214c04b11464e1ca841e97; no voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4
I20260924 03:11:02.692715 15679 raft_consensus.cc:2726] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:03.316514 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:03.316315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:33429)
0924 03:11:03.316434 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:03.316467 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260924 03:11:03.359710 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:03.359540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:57143)
0924 03:11:03.359651 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:03.359679 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260924 03:11:03.480238 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:03.480075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:52027)
0924 03:11:03.480177 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:03.480207 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260924 03:11:04.317194 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:04.317001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:36465)
0924 03:11:04.317123 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:04.317150 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 03:11:04.360376 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:04.360111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:44709)
0924 03:11:04.360303 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:04.360334 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":60,"threads_started":1}
I20260924 03:11:04.480921 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:04.480713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:59149)
0924 03:11:04.480834 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:04.480868 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260924 03:11:05.317932 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:05.317698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:40963)
0924 03:11:05.317835 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:05.317879 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260924 03:11:05.361089 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:05.360919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:58195)
0924 03:11:05.361030 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:05.361061 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260924 03:11:05.481488 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:05.481281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:54961)
0924 03:11:05.481427 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:05.481456 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260924 03:11:06.318735 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:06.318509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:50263)
0924 03:11:06.318653 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:06.318689 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260924 03:11:06.361697 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:06.361499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:50909)
0924 03:11:06.361631 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:06.361664 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260924 03:11:06.482035 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:06.481849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:41397)
0924 03:11:06.481969 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:06.482000 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260924 03:11:07.030359 15692 raft_consensus.cc:482] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:07.030442 15692 raft_consensus.cc:504] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1934512a46c5488db4ddffb2d3a1e3ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45833 } } peers { permanent_uuid: "a2d305973b214c04b11464e1ca841e97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36513 } } peers { permanent_uuid: "b68866397240470fad4d46f6e128a4d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260924 03:11:07.030557 15692 leader_election.cc:288] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833), b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255)
W20260924 03:11:07.030942 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1934512a46c5488db4ddffb2d3a1e3ae (127.0.98.2:45833): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
W20260924 03:11:07.031006 15275 leader_election.cc:341] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b68866397240470fad4d46f6e128a4d4 (127.0.98.3:33255): Not found: Tablet not found: bcdcaebb07d645c1b85499c8635b99ef
I20260924 03:11:07.031023 15275 leader_election.cc:302] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2d305973b214c04b11464e1ca841e97; no voters: 1934512a46c5488db4ddffb2d3a1e3ae, b68866397240470fad4d46f6e128a4d4
I20260924 03:11:07.031126 15692 raft_consensus.cc:2726] T bcdcaebb07d645c1b85499c8635b99ef P a2d305973b214c04b11464e1ca841e97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:07.319422 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:07.319201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:49457)
0924 03:11:07.319342 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:07.319377 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260924 03:11:07.362293 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:07.362088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:52439)
0924 03:11:07.362222 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:07.362259 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260924 03:11:07.482611 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:07.482406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:42477)
0924 03:11:07.482541 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:07.482573 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260924 03:11:08.320046 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:08.319852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:54837)
0924 03:11:08.319965 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:08.319996 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260924 03:11:08.362845 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:08.362684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:53735)
0924 03:11:08.362785 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:08.362812 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260924 03:11:08.483186 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:08.482993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:53009)
0924 03:11:08.483115 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:08.483148 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260924 03:11:09.320874 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:09.320630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:59081)
0924 03:11:09.320783 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:09.320827 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260924 03:11:09.363622 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:09.363406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:52791)
0924 03:11:09.363524 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:09.363566 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 03:11:09.483937 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:09.483652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:38195)
0924 03:11:09.483771 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:09.483888 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260924 03:11:10.321758 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:10.321522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:47419)
0924 03:11:10.321673 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:10.321711 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260924 03:11:10.364329 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:10.364122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:43327)
0924 03:11:10.364264 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:10.364298 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 03:11:10.484575 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:10.484387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:58099)
0924 03:11:10.484504 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:10.484542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260924 03:11:11.322501 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:11.322290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:43331)
0924 03:11:11.322426 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:11.322455 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 03:11:11.365042 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:11.364836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:46543)
0924 03:11:11.364967 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:11.364998 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260924 03:11:11.485203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15226
I20260924 03:11:11.486331 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:11.485030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:46793)
0924 03:11:11.485233 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:11.485266 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":61,"threads_started":1}
I20260924 03:11:11.489862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15390
I20260924 03:11:11.493744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15523
2026-09-24T03:11:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18008 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:11:11 dist-test-slave-jtnz krb5kdc[15710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:11:11 dist-test-slave-jtnz krb5kdc[15710](info): set up 2 sockets
Sep 24 03:11:11 dist-test-slave-jtnz krb5kdc[15710](info): commencing operation
krb5kdc: starting...
W20260924 03:11:12.462075 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:11:12.569701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:12.644248 15723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:12.644456 15723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:12.644491 15723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:12.645519 15723 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:11:12.645572 15723 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:11:12.645591 15723 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:11:12.645607 15723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:11:12.645634 15723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:12.645655 15723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:12.645681 15723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:11:12.645704 15723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:11:12.646942 15723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:12.647174 15723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:12.647372 15723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:12.651752 15729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:12.651804 15723 server_base.cc:1034] running on GCE node
W20260924 03:11:12.651774 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:12.651774 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:12.652149 15723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:12.652418 15723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:12.653548 15723 hybrid_clock.cc:648] HybridClock initialized: now 1790219472653538 us; error 29 us; skew 500 ppm
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:12.655936 15723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:11:12.656309 15723 webserver.cc:460] Webserver started at http://127.0.98.62:33889/ using document root <none> and password file <none>
I20260924 03:11:12.656534 15723 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:12.656574 15723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:12.656754 15723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:11:12.657621 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "3568def735cc4673950ed8415c531ce5"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "5503dfed6de81880c03361175ff9c883"
server_key_iv: "14bae06a05bbfa169cc8dac5e7d1ed9f"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.657966 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "3568def735cc4673950ed8415c531ce5"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "5503dfed6de81880c03361175ff9c883"
server_key_iv: "14bae06a05bbfa169cc8dac5e7d1ed9f"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.659289 15723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:11:12.659938 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:12.660105 15723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:11:12.660187 15723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "3568def735cc4673950ed8415c531ce5"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "5503dfed6de81880c03361175ff9c883"
server_key_iv: "14bae06a05bbfa169cc8dac5e7d1ed9f"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.660254 15723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:12.670704 15723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:12.672168 15723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:12.672302 15723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:12.675839 15723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46483
I20260924 03:11:12.675887 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46483 every 8 connection(s)
I20260924 03:11:12.676136 15723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:11:12.676838 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:12.678699 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap starting.
I20260924 03:11:12.679209 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:12.679477 15791 log.cc:824] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:12.680034 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: No bootstrap required, opened a new log
I20260924 03:11:12.681438 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:12.681557 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:12.681578 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Initialized, Role: FOLLOWER
I20260924 03:11:12.681733 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:12.681800 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:11:12.681844 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:11:12.681892 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:12.682426 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:12.682528 15791 leader_election.cc:302] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3568def735cc4673950ed8415c531ce5; no voters:
I20260924 03:11:12.682711 15791 leader_election.cc:288] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:11:12.682757 15794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:11:12.682928 15794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Running, Role: LEADER
I20260924 03:11:12.682934 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:11:12.683012 15794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:12.683319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15723
I20260924 03:11:12.683420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:11:12.683624 392 external_mini_cluster.cc:1428] Setting key 7f29f5c747c232aaea194b3d75d3e2a9
I20260924 03:11:12.683827 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3568def735cc4673950ed8415c531ce5. Latest consensus state: current_term: 1 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:12.683908 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:12.684216 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:11:12.684365 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:12.684478 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:12.684726 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:11:12.686841 15805 catalog_manager.cc:1347] Generated new cluster ID: efdb5a97fe5f47ad93d2a1863215e03e
I20260924 03:11:12.686898 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:12.693907 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:12.684053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.98.62:46483)
0924 03:11:12.684504 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:12.684513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:12.684694 (+ 181us) server_negotiation.cc:407] Connection header received
0924 03:11:12.685014 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:12.685021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:12.685074 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:12.685223 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:12.685870 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.686748 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.687698 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.688227 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.688315 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:12.690752 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:12.690757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:12.690757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:12.690770 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:12.692554 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:12.692882 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:12.692885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:12.692885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:12.692976 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:12.693146 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:12.693147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:12.693147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:12.693400 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:12.693517 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:12.693608 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:11:12.693720 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":58,"threads_started":1}
I20260924 03:11:12.706267 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:11:12.706681 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:11:12.713869 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Generated new TSK 0
I20260924 03:11:12.714046 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:11:12.740401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:12.816277 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:12.816437 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:12.816455 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:12.817607 15817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:12.817655 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:12.817679 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:11:12.819010 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:12.819175 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:12.819398 15817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:12.823661 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:12.823669 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:12.823701 15817 server_base.cc:1034] running on GCE node
W20260924 03:11:12.823702 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:12.824043 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:12.824267 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:12.825413 15817 hybrid_clock.cc:648] HybridClock initialized: now 1790219472825410 us; error 33 us; skew 500 ppm
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:12.827494 15817 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:11:12.827832 15817 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260924 03:11:12.828028 15817 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:12.828085 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:12.828212 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:11:12.829073 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "07840626c95143278f76942e16f7a846"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "563671654869596aca99a52619e66f5a"
server_key_iv: "6299c4d66e2b8d0db57a06d1c5db3806"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.829406 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07840626c95143278f76942e16f7a846"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "563671654869596aca99a52619e66f5a"
server_key_iv: "6299c4d66e2b8d0db57a06d1c5db3806"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.830551 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:11:12.831198 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:12.831336 15817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:11:12.831423 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "07840626c95143278f76942e16f7a846"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "563671654869596aca99a52619e66f5a"
server_key_iv: "6299c4d66e2b8d0db57a06d1c5db3806"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.831496 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:12.868127 15817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:12.869740 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:12.869894 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:12.870091 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:12.870396 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:12.870451 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:12.870489 15817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:12.870517 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:12.876106 15817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37467
I20260924 03:11:12.876139 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37467 every 8 connection(s)
I20260924 03:11:12.876415 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:12.884579 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:12.877057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42411 (local address 127.0.98.62:46483)
0924 03:11:12.877130 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:12.877132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:12.877283 (+ 151us) server_negotiation.cc:407] Connection header received
0924 03:11:12.877683 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:12.877683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:12.877697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:12.877743 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:12.878561 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.879043 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.879954 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.880171 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.880216 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:12.882046 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:12.882047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:12.882048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:12.882054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:12.883352 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:12.883749 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:12.883750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:12.883750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:12.883821 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:12.884045 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:12.884045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:12.884045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:12.884146 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:12.884199 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:12.884423 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 03:11:12.884487 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:11:12.884660 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:12.877008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:42411)
0924 03:11:12.877176 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:12.877207 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:12.877611 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:12.877787 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:12.877789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:12.878323 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:12.878487 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:12.878492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.879113 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.879113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:12.879871 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:12.879873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:12.880233 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:12.880234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:12.880333 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:12.880702 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:12.880709 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:12.881966 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:12.883452 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:12.883458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:12.883459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:12.883674 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:12.883872 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:12.883873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:12.883873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:12.883988 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:12.884202 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:12.884206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:12.884319 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 03:11:12.884394 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:11:12.884499 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 03:11:12.884810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15817
I20260924 03:11:12.884938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:11:12.885136 392 external_mini_cluster.cc:1428] Setting key 7c1c5b4f62437340e0b38f0c33cc4570
I20260924 03:11:12.885138 15946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:12.885211 15946 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:12.885370 15946 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:12.885874 15755 ts_manager.cc:194] Registered new tserver with Master: 07840626c95143278f76942e16f7a846 (127.0.98.1:37467)
I20260924 03:11:12.886472 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42411
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:11:12.906576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:12.978961 15951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:12.979110 15951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:12.979130 15951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:12.980073 15951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:12.980103 15951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:12.980141 15951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:11:12.981374 15951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:12.981536 15951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:12.981698 15951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:12.986239 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:12.986249 15956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:12.986248 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:12.986341 15951 server_base.cc:1034] running on GCE node
I20260924 03:11:12.986584 15951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:12.986804 15951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:12.987942 15951 hybrid_clock.cc:648] HybridClock initialized: now 1790219472987915 us; error 52 us; skew 500 ppm
Sep 24 03:11:12 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:12.989998 15951 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:11:12.990408 15951 webserver.cc:460] Webserver started at http://127.0.98.2:37067/ using document root <none> and password file <none>
I20260924 03:11:12.990613 15951 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:12.990674 15951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:12.990794 15951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:11:12.991667 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "4c62310824d634302127d7cde17f249c"
server_key_iv: "ee8698357a26c20bb8b3b59c1c5915fc"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.991997 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "4c62310824d634302127d7cde17f249c"
server_key_iv: "ee8698357a26c20bb8b3b59c1c5915fc"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.993268 15951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:11:12.993933 15966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:12.994146 15951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:11:12.994256 15951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "4c62310824d634302127d7cde17f249c"
server_key_iv: "ee8698357a26c20bb8b3b59c1c5915fc"
server_key_version: "encryptionkey@0"
I20260924 03:11:12.994336 15951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:13.009218 15951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:13.010890 15951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:13.011027 15951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:13.011214 15951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:13.011475 15951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:13.011533 15951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:13.011559 15951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:13.011584 15951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:13.016633 15951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36421
I20260924 03:11:13.016714 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36421 every 8 connection(s)
I20260924 03:11:13.016952 15951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:11:13.020435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 15951
I20260924 03:11:13.020532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:11:13.020691 392 external_mini_cluster.cc:1428] Setting key 66481b220efc1e1a0b0dfde7cb550eb6
Sep 24 03:11:13 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:13.025650 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.017510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:54431)
0924 03:11:13.017652 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.017680 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.018041 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.018279 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.018283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.018777 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:13.018984 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.018988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.019603 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.019603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.020352 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.020354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.020829 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.020830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.020966 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.021440 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:13.021448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:13.022751 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:13.024427 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:13.024433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:13.024434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:13.024634 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:13.024859 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:13.024860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:13.024860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:13.024953 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:13.025259 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:13.025262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:13.025359 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.025429 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.025525 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260924 03:11:13.025677 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.017553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54431 (local address 127.0.98.62:46483)
0924 03:11:13.017626 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.017628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.017758 (+ 130us) server_negotiation.cc:407] Connection header received
0924 03:11:13.018112 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.018113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.018122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.018167 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:13.019060 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.019530 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.020433 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.020718 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.020783 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.022834 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:13.022836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:13.022836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:13.022843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:13.024349 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.024714 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.024715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.024716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.024779 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.025024 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.025025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.025026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.025171 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:13.025263 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.025477 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.025564 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:11:13.026098 16080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:13.026203 16080 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:13.026376 16080 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:13.026690 15755 ts_manager.cc:194] Registered new tserver with Master: 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:13.027012 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54431
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:11:13.042893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:13.114744 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:13.114926 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:13.114948 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:13.115990 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:13.116022 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:13.116045 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:11:13.117298 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:13.117450 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:13.117591 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:13.122210 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:13.122346 16085 server_base.cc:1034] running on GCE node
W20260924 03:11:13.122267 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:13.122208 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:13.122640 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:13.122810 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:13.123940 16085 hybrid_clock.cc:648] HybridClock initialized: now 1790219473123930 us; error 25 us; skew 500 ppm
Sep 24 03:11:13 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:13.126104 16085 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:11:13.126523 16085 webserver.cc:460] Webserver started at http://127.0.98.3:33999/ using document root <none> and password file <none>
I20260924 03:11:13.126726 16085 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:13.126786 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:13.126921 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:11:13.127729 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f74109c20ee460cbd9680363cdfbaf1"
format_stamp: "Formatted at 2026-09-24 03:11:13 on dist-test-slave-jtnz"
server_key: "4fb33f8d7adde8efb15a0a58102d3c1b"
server_key_iv: "4c62277e33b50ed03ed01854f0e3729f"
server_key_version: "encryptionkey@0"
I20260924 03:11:13.127986 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f74109c20ee460cbd9680363cdfbaf1"
format_stamp: "Formatted at 2026-09-24 03:11:13 on dist-test-slave-jtnz"
server_key: "4fb33f8d7adde8efb15a0a58102d3c1b"
server_key_iv: "4c62277e33b50ed03ed01854f0e3729f"
server_key_version: "encryptionkey@0"
I20260924 03:11:13.129092 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:11:13.129726 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:13.129920 16085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:11:13.130055 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "3f74109c20ee460cbd9680363cdfbaf1"
format_stamp: "Formatted at 2026-09-24 03:11:13 on dist-test-slave-jtnz"
server_key: "4fb33f8d7adde8efb15a0a58102d3c1b"
server_key_iv: "4c62277e33b50ed03ed01854f0e3729f"
server_key_version: "encryptionkey@0"
I20260924 03:11:13.130136 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:13.140978 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:13.142697 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:13.142848 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:13.143054 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:13.143339 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:13.143388 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:13.143414 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:13.143441 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:13.148940 16085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37357
I20260924 03:11:13.148981 16213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37357 every 8 connection(s)
I20260924 03:11:13.149269 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:11:13 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:13.156908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16085
I20260924 03:11:13.157019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:11:13.157191 392 external_mini_cluster.cc:1428] Setting key 659915a750f7c2c59b7020723a071631
I20260924 03:11:13.157608 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.149896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50711 (local address 127.0.98.62:46483)
0924 03:11:13.149955 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.149957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.150123 (+ 166us) server_negotiation.cc:407] Connection header received
0924 03:11:13.150576 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.150577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.150589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.150645 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:13.151422 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.151900 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.152803 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.153001 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.153091 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.155044 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:13.155045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:13.155045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:13.155053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:13.156213 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.156596 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.156597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.156598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.156655 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.156906 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.156907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.156907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.157046 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:13.157106 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.157400 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.157484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:11:13.157754 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.149879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:50711)
0924 03:11:13.150028 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.150055 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.150483 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.150700 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.150704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.151170 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:13.151323 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.151327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.151986 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.151987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.152720 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.152722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.153095 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.153096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.153211 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.153674 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:13.153681 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:13.154948 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:13.156305 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:13.156311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:13.156313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:13.156510 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:13.156721 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:13.156722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:13.156722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:13.156838 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:13.157098 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:13.157104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:13.157226 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.157343 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.157511 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260924 03:11:13.158300 16214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:13.158386 16214 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:13.158538 16214 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:13.158919 15755 ts_manager.cc:194] Registered new tserver with Master: 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357)
I20260924 03:11:13.159279 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50711
I20260924 03:11:13.169142 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:11:13.175772 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.170393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.0.98.62:46483)
0924 03:11:13.170456 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.170458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.170563 (+ 105us) server_negotiation.cc:407] Connection header received
0924 03:11:13.170618 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.170619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.170628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.170668 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:13.171179 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.171593 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.172267 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.172443 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.172508 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.173541 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:13.173542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:13.173543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:13.173549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:13.174686 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.175043 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.175044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.175045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.175101 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:13.175300 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:13.175301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:13.175301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:13.175429 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:13.175526 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.175620 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.175687 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:11:13.177093 15755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:11:13.178637 15755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:11:13.184614 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36421 (local address 127.0.0.1:39062)
0924 03:11:13.180628 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.180662 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.180723 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.180926 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.180928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.180931 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:13.180988 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.180990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182203 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.182204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.183479 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.183482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184345 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.184419 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184431 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.184501 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.184548 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
I20260924 03:11:13.184629 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.0.98.2:36421)
0924 03:11:13.180766 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.180771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.180780 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:11:13.180815 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.180817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.180831 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.180924 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:13.181043 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182080 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.183572 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184268 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184342 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.184517 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.184555 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260924 03:11:13.184903 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37467 (local address 127.0.0.1:48148)
0924 03:11:13.180897 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.180905 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.180979 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.181147 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.181148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.181150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:13.181179 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.181180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182541 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.182542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.183698 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.183700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184702 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.184752 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184755 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.184798 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.184827 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":49,"threads_started":1}
I20260924 03:11:13.184984 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.0.98.3:37357)
0924 03:11:13.181195 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.181198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.181207 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:11:13.181257 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.181258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.181290 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.181365 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:13.181466 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182479 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.183719 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184519 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184585 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184636 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.184869 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.184901 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":73,"threads_started":1}
I20260924 03:11:13.185261 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.0.98.1:37467)
0924 03:11:13.180850 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.180854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.180969 (+ 115us) server_negotiation.cc:407] Connection header received
0924 03:11:13.181046 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.181049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.181065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.181146 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:13.181245 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182399 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.183782 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184579 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184631 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184676 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.185144 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.185196 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260924 03:11:13.185086 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.180504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37357 (local address 127.0.0.1:38986)
0924 03:11:13.181052 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.181060 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.181096 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.181361 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.181362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.181363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:13.181398 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.181399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.182571 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.182572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.183644 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.183646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.184670 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.184671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.184730 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.184734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.184781 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.184833 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":19,"thread_start_us":34,"threads_started":1}
I20260924 03:11:13.185596 16014 tablet_service.cc:1467] Processing CreateTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 (DEFAULT_TABLE table=test-table [id=292d47320f7a41fab4ee4c0c6190da96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:13.185947 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418f46f6c91f4e4bb57d7169c3932806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:13.185971 15880 tablet_service.cc:1467] Processing CreateTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 (DEFAULT_TABLE table=test-table [id=292d47320f7a41fab4ee4c0c6190da96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:13.186077 16148 tablet_service.cc:1467] Processing CreateTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 (DEFAULT_TABLE table=test-table [id=292d47320f7a41fab4ee4c0c6190da96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:13.186232 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418f46f6c91f4e4bb57d7169c3932806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:13.186388 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418f46f6c91f4e4bb57d7169c3932806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:13.189123 16229 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Bootstrap starting.
I20260924 03:11:13.189301 16228 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Bootstrap starting.
I20260924 03:11:13.189399 16230 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Bootstrap starting.
I20260924 03:11:13.189719 16229 tablet_bootstrap.cc:654] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:13.189926 16230 tablet_bootstrap.cc:654] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:13.189926 16228 tablet_bootstrap.cc:654] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:13.190289 16229 log.cc:824] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:13.190480 16230 log.cc:824] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:13.190508 16228 log.cc:824] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:13.190917 16229 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: No bootstrap required, opened a new log
I20260924 03:11:13.190979 16229 ts_tablet_manager.cc:1397] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:11:13.191109 16230 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: No bootstrap required, opened a new log
I20260924 03:11:13.191121 16228 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: No bootstrap required, opened a new log
I20260924 03:11:13.191166 16228 ts_tablet_manager.cc:1397] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:11:13.191179 16230 ts_tablet_manager.cc:1397] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:11:13.192538 16230 raft_consensus.cc:348] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.192649 16230 raft_consensus.cc:374] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:13.192667 16230 raft_consensus.cc:729] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f74109c20ee460cbd9680363cdfbaf1, State: Initialized, Role: FOLLOWER
I20260924 03:11:13.192798 16230 consensus_queue.cc:260] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.192764 16229 raft_consensus.cc:348] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.192957 16229 raft_consensus.cc:374] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:13.192986 16229 raft_consensus.cc:729] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0f4a5c3d9742e38950a0400abd8e8e, State: Initialized, Role: FOLLOWER
I20260924 03:11:13.192922 16228 raft_consensus.cc:348] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.193073 16228 raft_consensus.cc:374] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:13.193077 16230 ts_tablet_manager.cc:1428] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:11:13.193105 16228 raft_consensus.cc:729] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07840626c95143278f76942e16f7a846, State: Initialized, Role: FOLLOWER
I20260924 03:11:13.193132 16229 consensus_queue.cc:260] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.193074 16214 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
I20260924 03:11:13.193192 16228 consensus_queue.cc:260] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.193432 16229 ts_tablet_manager.cc:1428] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:11:13.193426 16228 ts_tablet_manager.cc:1428] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:11:13.193529 15946 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
I20260924 03:11:13.193791 16080 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
W20260924 03:11:13.267426 16081 tablet.cc:2369] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:11:13.377162 15947 tablet.cc:2369] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:11:13.399797 16215 tablet.cc:2369] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:11:13.403421 16234 raft_consensus.cc:482] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:13.403550 16234 raft_consensus.cc:504] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.403964 16234 leader_election.cc:288] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07840626c95143278f76942e16f7a846 (127.0.98.1:37467), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:13.407678 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.404441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45693 (local address 127.0.98.1:37467)
0924 03:11:13.404491 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.404493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.404503 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:11:13.404529 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.404530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.404542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.404596 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:13.404746 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.405487 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.406897 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.407382 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.407445 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.407454 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.407588 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.407620 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:11:13.407678 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.404457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50359 (local address 127.0.98.2:36421)
0924 03:11:13.404499 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.404500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.404509 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:11:13.404604 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.404605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.404617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.404676 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:13.404856 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.405532 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.406898 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.407378 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.407445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.407454 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.407588 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.407620 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:11:13.407696 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.404152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37467 (local address 127.0.98.3:45693)
0924 03:11:13.404193 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.404217 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.404278 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.404608 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.404610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.404612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:13.404687 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.404689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.405579 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.405580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.406818 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.406821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.407450 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.407451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.407521 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.407522 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.407571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.407613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":56}
I20260924 03:11:13.407703 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.404289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36421 (local address 127.0.98.3:50359)
0924 03:11:13.404444 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.404455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:13.404512 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:13.404683 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:13.404684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:13.404686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:13.404762 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.404764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.405599 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.405599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.406819 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:13.406821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.407450 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.407451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:13.407517 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.407521 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:11:13.407571 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:11:13.407613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":70,"thread_start_us":98,"threads_started":1}
I20260924 03:11:13.408097 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418f46f6c91f4e4bb57d7169c3932806" candidate_uuid: "3f74109c20ee460cbd9680363cdfbaf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07840626c95143278f76942e16f7a846" is_pre_election: true
I20260924 03:11:13.408129 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418f46f6c91f4e4bb57d7169c3932806" candidate_uuid: "3f74109c20ee460cbd9680363cdfbaf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" is_pre_election: true
I20260924 03:11:13.408196 15900 raft_consensus.cc:2454] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f74109c20ee460cbd9680363cdfbaf1 in term 0.
I20260924 03:11:13.408298 16034 raft_consensus.cc:2454] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f74109c20ee460cbd9680363cdfbaf1 in term 0.
I20260924 03:11:13.408422 16102 leader_election.cc:302] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07840626c95143278f76942e16f7a846, 3f74109c20ee460cbd9680363cdfbaf1; no voters:
I20260924 03:11:13.408540 16234 raft_consensus.cc:2781] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:11:13.408617 16234 raft_consensus.cc:482] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:11:13.408645 16234 raft_consensus.cc:3037] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:13.409286 16234 raft_consensus.cc:504] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.409420 16234 leader_election.cc:288] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [CANDIDATE]: Term 1 election: Requested vote from peers 07840626c95143278f76942e16f7a846 (127.0.98.1:37467), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:13.409549 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418f46f6c91f4e4bb57d7169c3932806" candidate_uuid: "3f74109c20ee460cbd9680363cdfbaf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
I20260924 03:11:13.409564 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418f46f6c91f4e4bb57d7169c3932806" candidate_uuid: "3f74109c20ee460cbd9680363cdfbaf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07840626c95143278f76942e16f7a846"
I20260924 03:11:13.409616 16034 raft_consensus.cc:3037] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:13.409646 15900 raft_consensus.cc:3037] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:13.410310 16034 raft_consensus.cc:2454] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f74109c20ee460cbd9680363cdfbaf1 in term 1.
I20260924 03:11:13.410311 15900 raft_consensus.cc:2454] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f74109c20ee460cbd9680363cdfbaf1 in term 1.
I20260924 03:11:13.410447 16102 leader_election.cc:302] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07840626c95143278f76942e16f7a846, 3f74109c20ee460cbd9680363cdfbaf1; no voters:
I20260924 03:11:13.410555 16234 raft_consensus.cc:2781] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:11:13.410681 16234 raft_consensus.cc:686] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 LEADER]: Becoming Leader. State: Replica: 3f74109c20ee460cbd9680363cdfbaf1, State: Running, Role: LEADER
I20260924 03:11:13.410784 16234 consensus_queue.cc:237] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:13.411525 15753 catalog_manager.cc:5495] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3f74109c20ee460cbd9680363cdfbaf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } health_report { overall_health: HEALTHY } } }
I20260924 03:11:13.464473 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.462149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.0.98.3:37357)
0924 03:11:13.462208 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:13.462211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:13.462220 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:11:13.462316 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:13.462317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:13.462336 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:13.462428 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:11:13.462554 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.462997 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.463695 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:13.463889 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:13.463959 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:13.464135 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:11:13.464235 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:11:13.464321 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:13.464361 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 03:11:13.464379 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:11:13.466056 15900 raft_consensus.cc:1261] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Refusing update from remote peer 3f74109c20ee460cbd9680363cdfbaf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:13.466058 16034 raft_consensus.cc:1261] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Refusing update from remote peer 3f74109c20ee460cbd9680363cdfbaf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:13.466400 16234 consensus_queue.cc:1035] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:13.466548 16238 consensus_queue.cc:1035] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:13.468559 16241 mvcc.cc:205] Tried to move back new op lower bound from 7332738963314642944 to 7332738963091091456. Current Snapshot: MvccSnapshot[applied={T|T < 7332738963314642944}]
I20260924 03:11:13.469604 16242 mvcc.cc:205] Tried to move back new op lower bound from 7332738963314642944 to 7332738963091091456. Current Snapshot: MvccSnapshot[applied={T|T < 7332738963314642944 or (T in {7332738963314642944})}]
I20260924 03:11:13.470460 15753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:13.470554 15753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:13.471311 15753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Sending DeleteTablet for 3 replicas of tablet 418f46f6c91f4e4bb57d7169c3932806
I20260924 03:11:13.471664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15817
I20260924 03:11:13.471652 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39062
I20260924 03:11:13.471649 15880 tablet_service.cc:1514] Processing DeleteTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48148
I20260924 03:11:13.473158 16251 tablet_replica.cc:330] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: stopping tablet replica
I20260924 03:11:13.473258 16251 raft_consensus.cc:2229] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:13.473361 16251 raft_consensus.cc:2258] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:13.473583 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38986
I20260924 03:11:13.473824 16252 tablet_replica.cc:330] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: stopping tablet replica
I20260924 03:11:13.473834 16251 ts_tablet_manager.cc:1905] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:13.473964 16252 raft_consensus.cc:2229] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:11:13.474102 16252 raft_consensus.cc:2258] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:13.475929 16251 ts_tablet_manager.cc:1918] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:13.476018 16251 log.cc:1196] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/418f46f6c91f4e4bb57d7169c3932806
I20260924 03:11:13.476271 16251 ts_tablet_manager.cc:1939] T 418f46f6c91f4e4bb57d7169c3932806 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting consensus metadata
I20260924 03:11:13.476781 15742 catalog_manager.cc:4841] TS 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): tablet 418f46f6c91f4e4bb57d7169c3932806 (table test-table [id=292d47320f7a41fab4ee4c0c6190da96]) successfully deleted
I20260924 03:11:13.477031 16252 ts_tablet_manager.cc:1905] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:11:13.477932 15740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37467 (error 108)
I20260924 03:11:13.478204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15951
I20260924 03:11:13.478430 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:13.478315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37467 (local address 127.0.0.1:48156)
0924 03:11:13.478366 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:13.478395 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:11:13.478531 15740 catalog_manager.cc:4584] TS 07840626c95143278f76942e16f7a846 (127.0.98.1:37467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 418f46f6c91f4e4bb57d7169c3932806: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37467: connect: Connection refused (error 111)
I20260924 03:11:13.478577 16252 ts_tablet_manager.cc:1918] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:13.478633 16252 log.cc:1196] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/418f46f6c91f4e4bb57d7169c3932806
I20260924 03:11:13.478863 16252 ts_tablet_manager.cc:1939] T 418f46f6c91f4e4bb57d7169c3932806 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting consensus metadata
I20260924 03:11:13.479242 15741 catalog_manager.cc:4841] TS 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): tablet 418f46f6c91f4e4bb57d7169c3932806 (table test-table [id=292d47320f7a41fab4ee4c0c6190da96]) successfully deleted
I20260924 03:11:13.483911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16085
I20260924 03:11:13.488453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 15723
I20260924 03:11:33.493168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--webserver_interface=127.0.98.62
--webserver_port=33889
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:33.564945 16254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:33.565114 16254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:33.565132 16254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:33.566166 16254 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:11:33.566272 16254 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:11:33.566286 16254 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:11:33.566324 16254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:11:33.566350 16254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:33.566377 16254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:33.566404 16254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:11:33.566428 16254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:11:33.567754 16254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:33.567951 16254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:33.568142 16254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:33.575619 16260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.575649 16254 server_base.cc:1034] running on GCE node
W20260924 03:11:33.575624 16261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.575624 16263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.576117 16254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:33.576406 16254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:33.577534 16254 hybrid_clock.cc:648] HybridClock initialized: now 1790219493577524 us; error 26 us; skew 500 ppm
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:33.579773 16254 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:11:33.580090 16254 webserver.cc:460] Webserver started at http://127.0.98.62:33889/ using document root <none> and password file <none>
I20260924 03:11:33.580255 16254 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:33.580309 16254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:33.581466 16254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:33.582036 16270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.582198 16254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.582283 16254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "3568def735cc4673950ed8415c531ce5"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "5503dfed6de81880c03361175ff9c883"
server_key_iv: "14bae06a05bbfa169cc8dac5e7d1ed9f"
server_key_version: "encryptionkey@0"
I20260924 03:11:33.582587 16254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:33.592478 16254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:33.594121 16254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:33.594295 16254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:33.597780 16254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46483
I20260924 03:11:33.597802 16322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46483 every 8 connection(s)
I20260924 03:11:33.598150 16254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:11:33.599545 16323 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:11:33.600154 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap starting.
I20260924 03:11:33.607527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16254
I20260924 03:11:33.607777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37467
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=44573
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:33.608172 16323 log.cc:824] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:33.609886 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:11:33.610148 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap complete.
I20260924 03:11:33.612303 16323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:33.612540 16323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Initialized, Role: FOLLOWER
I20260924 03:11:33.612761 16323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:33.612843 16323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:11:33.612898 16323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:11:33.612957 16323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:11:33.613889 16323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:33.614001 16323 leader_election.cc:302] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3568def735cc4673950ed8415c531ce5; no voters:
I20260924 03:11:33.614179 16323 leader_election.cc:288] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:11:33.614238 16327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:11:33.614399 16327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 LEADER]: Becoming Leader. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Running, Role: LEADER
I20260924 03:11:33.614444 16323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:11:33.615752 16327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:33.616233 16328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3568def735cc4673950ed8415c531ce5. Latest consensus state: current_term: 2 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:33.616348 16329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:33.616438 16328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:33.616448 16329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:33.616858 16341 catalog_manager.cc:1259] Loaded cluster ID: efdb5a97fe5f47ad93d2a1863215e03e
I20260924 03:11:33.616912 16341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: loading cluster ID for follower catalog manager: success
I20260924 03:11:33.617678 16341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: acquiring CA information for follower catalog manager: success
I20260924 03:11:33.618016 16341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:11:33.618099 16343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:11:33.618484 16343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:11:33.618561 16343 catalog_manager.cc:1259] Loaded cluster ID: efdb5a97fe5f47ad93d2a1863215e03e
I20260924 03:11:33.618589 16343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:11:33.618731 16343 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:11:33.618804 16343 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Loaded TSK: 0
I20260924 03:11:33.633280 16343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Generated new TSK 1
I20260924 03:11:33.633358 16343 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:11:33.688211 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:33.688402 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:33.688424 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:33.689670 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:33.689713 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:33.689739 16326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:11:33.691270 16326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:33.691500 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:33.691738 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:33.696342 16351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.696349 16350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.696461 16353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.696727 16326 server_base.cc:1034] running on GCE node
I20260924 03:11:33.696943 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:33.697176 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:33.698310 16326 hybrid_clock.cc:648] HybridClock initialized: now 1790219493698297 us; error 30 us; skew 500 ppm
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:33.700631 16326 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:11:33.701023 16326 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260924 03:11:33.701267 16326 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:33.701323 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:33.702749 16326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:11:33.703383 16360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.703562 16326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:11:33.703651 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "07840626c95143278f76942e16f7a846"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "563671654869596aca99a52619e66f5a"
server_key_iv: "6299c4d66e2b8d0db57a06d1c5db3806"
server_key_version: "encryptionkey@0"
I20260924 03:11:33.703967 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:33.726322 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:33.728389 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:33.728540 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:33.728773 16326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:33.729223 16367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:11:33.730563 16326 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:11:33.730608 16326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:11:33.730640 16326 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:11:33.731246 16326 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:11:33.731298 16326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:11:33.731328 16367 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Bootstrap starting.
I20260924 03:11:33.738799 16326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37467
I20260924 03:11:33.738875 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37467 every 8 connection(s)
I20260924 03:11:33.739239 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:11:33.742040 16367 log.cc:824] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:33.742545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16326
I20260924 03:11:33.742828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36421
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=37067
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:33.744491 16367 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:11:33.744798 16367 tablet_bootstrap.cc:492] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Bootstrap complete.
I20260924 03:11:33.745834 16367 ts_tablet_manager.cc:1397] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:33.747401 16367 raft_consensus.cc:348] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:33.747756 16367 raft_consensus.cc:729] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07840626c95143278f76942e16f7a846, State: Initialized, Role: FOLLOWER
I20260924 03:11:33.747888 16367 consensus_queue.cc:260] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } }
I20260924 03:11:33.748171 16367 ts_tablet_manager.cc:1428] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
I20260924 03:11:33.749872 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.739905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:52309)
0924 03:11:33.740075 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:33.740113 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:33.740551 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:33.740939 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:33.740943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:33.741222 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:33.741397 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.741401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.742730 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.742731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.743573 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.743576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.744202 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.744204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.744388 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.744998 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:33.745007 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:33.746469 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:33.748427 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.748432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.748434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.748634 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.748871 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.748872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.748873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.748976 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.749317 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:33.749323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:33.749448 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 03:11:33.749546 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:11:33.749697 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260924 03:11:33.749872 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.740009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52309 (local address 127.0.98.62:46483)
0924 03:11:33.740185 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:33.740192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:33.740241 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:11:33.740738 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:33.740744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:33.740787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:33.740910 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:33.741507 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.742572 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.743658 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.744079 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.744633 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.746570 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:33.746574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:33.746574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:33.746585 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:33.748346 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.748710 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.748713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.748714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.748807 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.749044 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.749045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.749045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.749251 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:33.749339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:33.749595 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 03:11:33.749705 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260924 03:11:33.750613 16475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:33.750712 16475 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:33.750886 16475 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:33.751638 16287 ts_manager.cc:194] Registered new tserver with Master: 07840626c95143278f76942e16f7a846 (127.0.98.1:37467)
I20260924 03:11:33.752867 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52309
I20260924 03:11:33.753887 16475 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:11:33.759495 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.752265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.1:37467)
0924 03:11:33.752560 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:33.752563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:33.752569 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:11:33.752590 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:33.752591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:33.752600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:33.752672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:33.753040 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.753906 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.754765 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.755025 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.755123 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.756893 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:33.756900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:33.756900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:33.756907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:33.758276 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.758647 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.758649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.758650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.758704 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.758902 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.758904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.758904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.759050 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:33.759131 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:33.759335 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:11:33.759414 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":58,"threads_started":1}
I20260924 03:11:33.759493 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.752215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37467 (local address 127.0.0.1:42158)
0924 03:11:33.752409 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:33.752429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:33.752474 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:33.752675 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:33.752680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:33.752896 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:33.752953 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.752955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.753992 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.753993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.754669 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.754671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.755120 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.755124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.755194 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.755615 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:33.755619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:33.756807 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:33.758359 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.758365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.758367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.758540 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.758783 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.758784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.758785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.758837 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.759125 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:33.759130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:33.759178 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 03:11:33.759287 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:11:33.759400 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260924 03:11:33.759845 16409 tablet_service.cc:1514] Processing DeleteTablet for tablet 418f46f6c91f4e4bb57d7169c3932806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42158
I20260924 03:11:33.760252 16484 tablet_replica.cc:330] stopping tablet replica
I20260924 03:11:33.760352 16484 raft_consensus.cc:2229] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:33.760408 16484 raft_consensus.cc:2258] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:33.760658 16484 ts_tablet_manager.cc:1905] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:33.762912 16484 ts_tablet_manager.cc:1918] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:33.762980 16484 log.cc:1196] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal/wals/418f46f6c91f4e4bb57d7169c3932806
I20260924 03:11:33.763196 16484 ts_tablet_manager.cc:1939] T 418f46f6c91f4e4bb57d7169c3932806 P 07840626c95143278f76942e16f7a846: Deleting consensus metadata
I20260924 03:11:33.763736 16272 catalog_manager.cc:4841] TS 07840626c95143278f76942e16f7a846 (127.0.98.1:37467): tablet 418f46f6c91f4e4bb57d7169c3932806 (table test-table [id=292d47320f7a41fab4ee4c0c6190da96]) successfully deleted
W20260924 03:11:33.821394 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:33.821566 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:33.821588 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:33.822690 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:33.822724 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:33.822747 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:11:33.824018 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37067
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:33.824221 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:33.824416 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:11:33.828929 16480 server_base.cc:1034] running on GCE node
W20260924 03:11:33.828904 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.828918 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.829012 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.829329 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:33.829591 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:33.830737 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790219493830734 us; error 29 us; skew 500 ppm
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:33.832832 16480 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:11:33.833180 16480 webserver.cc:460] Webserver started at http://127.0.98.2:37067/ using document root <none> and password file <none>
I20260924 03:11:33.833377 16480 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:33.833443 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:33.834564 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:11:33.835016 16499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.835165 16480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.835247 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "4c62310824d634302127d7cde17f249c"
server_key_iv: "ee8698357a26c20bb8b3b59c1c5915fc"
server_key_version: "encryptionkey@0"
I20260924 03:11:33.835489 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:33.851394 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:33.852855 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:33.853006 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:33.853178 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:33.853428 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:33.853457 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.853474 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:33.853516 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.858847 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36421
I20260924 03:11:33.858911 16612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36421 every 8 connection(s)
I20260924 03:11:33.859201 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:33.867035 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.859824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58977 (local address 127.0.98.62:46483)
0924 03:11:33.859913 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:33.859915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:33.860025 (+ 110us) server_negotiation.cc:407] Connection header received
0924 03:11:33.860406 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:33.860406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:33.860415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:33.860461 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:33.861254 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.861709 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.862615 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.862808 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.862859 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.864606 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:33.864607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:33.864607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:33.864613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:33.865924 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.866348 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.866348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.866349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.866396 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.866590 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.866591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.866591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.866683 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:33.866730 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:33.866918 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 03:11:33.866975 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:11:33.867235 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.859782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:58977)
0924 03:11:33.859928 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:33.859956 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:33.860337 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:33.860505 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:33.860507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:33.861003 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:33.861175 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.861179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.861776 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.861777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.862548 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.862550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.862876 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.862877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.862965 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.863330 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:33.863338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:33.864530 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:33.865988 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.865992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.865994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.866288 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.866453 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.866454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.866454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.866541 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.866734 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:33.866738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:33.866835 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 03:11:33.866908 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:11:33.867007 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260924 03:11:33.867753 16613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:33.867833 16613 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:33.867997 16613 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:33.868063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16480
I20260924 03:11:33.868327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37357
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=33999
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:33.868355 16287 ts_manager.cc:194] Registered new tserver with Master: 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:33.868803 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58977
W20260924 03:11:33.942732 16616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:33.942967 16616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:33.943009 16616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:33.944159 16616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:33.944228 16616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:33.944278 16616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:11:33.945787 16616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33999
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:33.946053 16616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:33.946322 16616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:33.950845 16624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:33.950856 16622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.950971 16616 server_base.cc:1034] running on GCE node
W20260924 03:11:33.950871 16621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:33.951200 16616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:33.951385 16616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:33.952538 16616 hybrid_clock.cc:648] HybridClock initialized: now 1790219493952526 us; error 26 us; skew 500 ppm
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:33.954775 16616 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:11:33.955088 16616 webserver.cc:460] Webserver started at http://127.0.98.3:33999/ using document root <none> and password file <none>
I20260924 03:11:33.955255 16616 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:33.955297 16616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:33.956372 16616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:33.956898 16631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.957057 16616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.957146 16616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "3f74109c20ee460cbd9680363cdfbaf1"
format_stamp: "Formatted at 2026-09-24 03:11:13 on dist-test-slave-jtnz"
server_key: "4fb33f8d7adde8efb15a0a58102d3c1b"
server_key_iv: "4c62277e33b50ed03ed01854f0e3729f"
server_key_version: "encryptionkey@0"
I20260924 03:11:33.957443 16616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:33.975026 16616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:33.976590 16616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:33.976734 16616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:33.976928 16616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:33.977195 16616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:33.977245 16616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.977283 16616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:33.977310 16616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:33.982661 16616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37357
I20260924 03:11:33.982702 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37357 every 8 connection(s)
I20260924 03:11:33.982966 16616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:11:33.983435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16616
Sep 24 03:11:33 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:11:33.990463 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.983767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.0.98.62:46483)
0924 03:11:33.983946 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:33.983948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:33.983974 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:11:33.984087 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:33.984087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:33.984100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:33.984164 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:33.984935 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.985435 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.986287 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.986478 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.986528 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.987804 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:33.987805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:33.987805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:33.987810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:33.989187 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.989614 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.989615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.989615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.989678 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.989894 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.989895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.989896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.989997 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:33.990148 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:33.990278 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 03:11:33.990363 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":47,"threads_started":1}
I20260924 03:11:33.991976 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.983603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59197 (local address 127.0.98.62:46483)
0924 03:11:33.983659 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:33.983662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:33.983958 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:11:33.984543 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:33.984543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:33.984550 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:33.984582 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:33.985478 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.985928 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.987281 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.987489 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.987537 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.989392 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:33.989393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:33.989393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:33.989399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:33.990972 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.991353 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.991354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.991354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.991396 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:33.991579 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:33.991579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:33.991579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:33.991656 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:33.991698 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:33.991878 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 03:11:33.991925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:11:33.992120 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:33.983533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:59197)
0924 03:11:33.983706 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:33.983854 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:33.984455 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:33.984659 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:33.984662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:33.985241 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:33.985391 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.985395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.986014 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.986014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.987190 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:33.987193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:33.987567 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:33.987568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:33.987699 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:33.988140 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:33.988147 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:33.989319 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:33.991038 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.991044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.991045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.991293 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.991449 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:33.991450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:33.991450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:33.991530 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:33.991704 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:33.991707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:33.991799 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 03:11:33.991866 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:11:33.991959 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260924 03:11:33.992498 16745 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46483
I20260924 03:11:33.992584 16745 heartbeater.cc:464] Registering TS with master...
I20260924 03:11:33.992724 16745 heartbeater.cc:511] Master 127.0.98.62:46483 requested a full tablet report, sending...
I20260924 03:11:33.993049 16287 ts_manager.cc:194] Registered new tserver with Master: 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357)
I20260924 03:11:33.993371 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59197
I20260924 03:11:34.001950 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:11:34.004448 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.002326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.0.98.62:46483)
0924 03:11:34.002357 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.002359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.002507 (+ 148us) server_negotiation.cc:407] Connection header received
0924 03:11:34.002581 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.002582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.002623 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.002695 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:11:34.002795 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.003231 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.003888 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.004040 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.004097 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.004223 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:11:34.004270 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:11:34.004392 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52160: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:11:34.004493 16478 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52160: token expired
W20260924 03:11:34.004570 16218 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46483)
I20260924 03:11:34.009508 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.004803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.0.98.62:46483)
0924 03:11:34.004834 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.004835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.004842 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:11:34.004858 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.004859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.004865 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.004901 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:34.005283 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.005648 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.006288 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.006525 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.006602 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.007563 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:34.007565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:34.007565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:34.007571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:34.008527 (+ 956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:34.008835 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:34.008836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:34.008836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:34.008883 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:34.009074 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:34.009075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:34.009075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:34.009166 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:34.009260 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.009340 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.009423 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:11:34.010439 16287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:11:34.011905 16287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:11:34.013764 16409 tablet_service.cc:1467] Processing CreateTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 (DEFAULT_TABLE table=test-table [id=e22880e9fbbb4cc6a77a40899dba1aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:34.013967 16409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dc231796c94ac7963dc2d2adc722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:34.015476 16367 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Bootstrap starting.
I20260924 03:11:34.015779 16367 tablet_bootstrap.cc:654] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:34.016273 16367 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: No bootstrap required, opened a new log
I20260924 03:11:34.016319 16367 ts_tablet_manager.cc:1397] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:11:34.016661 16367 raft_consensus.cc:348] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.016745 16367 raft_consensus.cc:374] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:34.016810 16367 raft_consensus.cc:729] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07840626c95143278f76942e16f7a846, State: Initialized, Role: FOLLOWER
I20260924 03:11:34.016865 16367 consensus_queue.cc:260] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.016965 16367 ts_tablet_manager.cc:1428] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:11:34.017287 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.013553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37357 (local address 127.0.0.1:53132)
0924 03:11:34.013720 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:34.013727 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:34.013772 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:34.014426 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:34.014427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:34.014429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:34.014466 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.014468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.015408 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.015409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.016248 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.016251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.017053 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.017054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.017125 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.017147 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:11:34.017177 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 03:11:34.017215 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
I20260924 03:11:34.017606 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.013687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.98.3:37357)
0924 03:11:34.014278 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.014281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.014287 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:11:34.014329 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.014330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.014341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.014416 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:34.014528 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.015338 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.016334 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.016954 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.017392 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.017452 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.017476 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.017518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":78,"threads_started":1}
I20260924 03:11:34.017781 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.013553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36421 (local address 127.0.0.1:34252)
0924 03:11:34.013601 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:34.013610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:34.013645 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:34.014034 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:34.014035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:34.014037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:34.014071 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.014073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.015306 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.015307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.016247 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.016249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.017551 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.017552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.017615 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.017618 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:11:34.017668 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:11:34.017712 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:11:34.017910 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.013719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.98.2:36421)
0924 03:11:34.013873 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.013877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.013887 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:11:34.013922 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.013924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.013938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.014014 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:34.014128 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.015243 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.016400 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.017480 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.017551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.017598 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.017782 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.017822 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260924 03:11:34.018266 16679 tablet_service.cc:1467] Processing CreateTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 (DEFAULT_TABLE table=test-table [id=e22880e9fbbb4cc6a77a40899dba1aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:34.018457 16547 tablet_service.cc:1467] Processing CreateTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 (DEFAULT_TABLE table=test-table [id=e22880e9fbbb4cc6a77a40899dba1aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:11:34.018561 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dc231796c94ac7963dc2d2adc722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:34.018685 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dc231796c94ac7963dc2d2adc722f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:11:34.021003 16755 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Bootstrap starting.
I20260924 03:11:34.021164 16756 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Bootstrap starting.
I20260924 03:11:34.021724 16755 tablet_bootstrap.cc:654] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:34.021801 16756 tablet_bootstrap.cc:654] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Neither blocks nor log segments found. Creating new log.
I20260924 03:11:34.022408 16756 log.cc:824] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:34.022397 16755 log.cc:824] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:34.023053 16756 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: No bootstrap required, opened a new log
I20260924 03:11:34.023113 16756 ts_tablet_manager.cc:1397] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:11:34.023211 16755 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: No bootstrap required, opened a new log
I20260924 03:11:34.023269 16755 ts_tablet_manager.cc:1397] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:11:34.024926 16756 raft_consensus.cc:348] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.024996 16755 raft_consensus.cc:348] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.025101 16756 raft_consensus.cc:374] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:34.025113 16755 raft_consensus.cc:374] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:11:34.025146 16756 raft_consensus.cc:729] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0f4a5c3d9742e38950a0400abd8e8e, State: Initialized, Role: FOLLOWER
I20260924 03:11:34.025146 16755 raft_consensus.cc:729] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f74109c20ee460cbd9680363cdfbaf1, State: Initialized, Role: FOLLOWER
I20260924 03:11:34.025251 16756 consensus_queue.cc:260] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.025271 16755 consensus_queue.cc:260] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.025501 16756 ts_tablet_manager.cc:1428] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:11:34.025501 16755 ts_tablet_manager.cc:1428] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:11:34.025516 16613 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
I20260924 03:11:34.025597 16745 heartbeater.cc:503] Master 127.0.98.62:46483 was elected leader, sending a full tablet report...
W20260924 03:11:34.109717 16614 tablet.cc:2369] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:11:34.121623 16481 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:34.121788 16481 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.122104 16481 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:34.125658 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.122489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50389 (local address 127.0.98.2:36421)
0924 03:11:34.122550 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.122552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.122633 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:11:34.122659 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.122660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.122670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.122721 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:34.122854 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.123471 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.124795 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.125322 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.125387 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.125396 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.125555 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.125599 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:11:34.125658 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.122389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59881 (local address 127.0.98.3:37357)
0924 03:11:34.122470 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.122472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.122483 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:11:34.122511 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.122512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.122524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.122600 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:11:34.122787 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.123471 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.124795 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.125317 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.125387 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.125395 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.125555 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.125600 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:11:34.125680 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.122284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37357 (local address 127.0.98.1:59881)
0924 03:11:34.122347 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:34.122360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:34.122421 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:34.122616 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:34.122618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:34.122620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:34.122692 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.122694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.123545 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.123546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.124712 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.124715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.125392 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.125393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.125460 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.125484 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:11:34.125538 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 03:11:34.125598 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":41}
I20260924 03:11:34.125694 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.122296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36421 (local address 127.0.98.1:50389)
0924 03:11:34.122551 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:34.122565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:34.122622 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:34.122732 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:34.122733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:34.122734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:11:34.122796 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.122797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.123545 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.123546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.124712 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:34.124715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.125409 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.125409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:34.125460 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.125484 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:11:34.125538 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 03:11:34.125598 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":127,"thread_start_us":116,"threads_started":1}
I20260924 03:11:34.126091 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dc231796c94ac7963dc2d2adc722f0" candidate_uuid: "07840626c95143278f76942e16f7a846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f74109c20ee460cbd9680363cdfbaf1" is_pre_election: true
I20260924 03:11:34.126161 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dc231796c94ac7963dc2d2adc722f0" candidate_uuid: "07840626c95143278f76942e16f7a846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" is_pre_election: true
I20260924 03:11:34.126276 16699 raft_consensus.cc:2454] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07840626c95143278f76942e16f7a846 in term 0.
I20260924 03:11:34.126368 16567 raft_consensus.cc:2454] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07840626c95143278f76942e16f7a846 in term 0.
I20260924 03:11:34.126518 16363 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07840626c95143278f76942e16f7a846, 3f74109c20ee460cbd9680363cdfbaf1; no voters:
I20260924 03:11:34.126657 16481 raft_consensus.cc:2781] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:11:34.126714 16481 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:11:34.126736 16481 raft_consensus.cc:3037] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:34.127561 16481 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.127708 16481 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 1 election: Requested vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
I20260924 03:11:34.127836 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dc231796c94ac7963dc2d2adc722f0" candidate_uuid: "07840626c95143278f76942e16f7a846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
I20260924 03:11:34.127836 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dc231796c94ac7963dc2d2adc722f0" candidate_uuid: "07840626c95143278f76942e16f7a846" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f74109c20ee460cbd9680363cdfbaf1"
I20260924 03:11:34.127912 16567 raft_consensus.cc:3037] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:34.127912 16699 raft_consensus.cc:3037] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:11:34.128755 16567 raft_consensus.cc:2454] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07840626c95143278f76942e16f7a846 in term 1.
I20260924 03:11:34.128755 16699 raft_consensus.cc:2454] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07840626c95143278f76942e16f7a846 in term 1.
I20260924 03:11:34.128954 16363 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07840626c95143278f76942e16f7a846, 3f74109c20ee460cbd9680363cdfbaf1; no voters:
I20260924 03:11:34.129050 16481 raft_consensus.cc:2781] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:11:34.129169 16481 raft_consensus.cc:686] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 LEADER]: Becoming Leader. State: Replica: 07840626c95143278f76942e16f7a846, State: Running, Role: LEADER
I20260924 03:11:34.129251 16481 consensus_queue.cc:237] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:34.129931 16287 catalog_manager.cc:5495] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07840626c95143278f76942e16f7a846 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "07840626c95143278f76942e16f7a846" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } health_report { overall_health: HEALTHY } } }
I20260924 03:11:34.159942 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.157863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.0.98.1:37467)
0924 03:11:34.157912 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:34.157914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:34.157923 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:11:34.157984 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:34.157985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:34.158004 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:34.158069 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:11:34.158171 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.158603 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.159249 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:34.159432 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:34.159497 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:34.159641 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:11:34.159744 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:11:34.159815 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:34.159853 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:11:34.159871 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:11:34.161212 16567 raft_consensus.cc:1261] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Refusing update from remote peer 07840626c95143278f76942e16f7a846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:34.161212 16699 raft_consensus.cc:1261] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Refusing update from remote peer 07840626c95143278f76942e16f7a846: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:11:34.161543 16481 consensus_queue.cc:1035] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:34.161715 16762 consensus_queue.cc:1035] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:11:34.163786 16766 mvcc.cc:205] Tried to move back new op lower bound from 7332739048082100224 to 7332739047953801216. Current Snapshot: MvccSnapshot[applied={T|T < 7332739048082100224}]
I20260924 03:11:34.165462 16287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:34.165548 16287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:11:34.166128 16287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Sending DeleteTablet for 3 replicas of tablet a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:34.166435 16409 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42158
I20260924 03:11:34.166460 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34252
I20260924 03:11:34.166489 16679 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dc231796c94ac7963dc2d2adc722f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:11:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260924 03:11:34.166615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16326
I20260924 03:11:34.166621 16484 tablet_replica.cc:330] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: stopping tablet replica
I20260924 03:11:34.166671 16484 raft_consensus.cc:2229] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:11:34.166734 16484 raft_consensus.cc:2258] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:34.166939 16775 tablet_replica.cc:330] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: stopping tablet replica
I20260924 03:11:34.167097 16775 raft_consensus.cc:2229] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:34.167229 16775 raft_consensus.cc:2258] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:34.167675 16775 ts_tablet_manager.cc:1905] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:34.168075 16776 tablet_replica.cc:330] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: stopping tablet replica
I20260924 03:11:34.168174 16776 raft_consensus.cc:2229] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:11:34.168265 16776 raft_consensus.cc:2258] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:11:34.168788 16776 ts_tablet_manager.cc:1905] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:11:34.169625 16775 ts_tablet_manager.cc:1918] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:34.169692 16775 log.cc:1196] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:34.169966 16775 ts_tablet_manager.cc:1939] T a4dc231796c94ac7963dc2d2adc722f0 P 2d0f4a5c3d9742e38950a0400abd8e8e: Deleting consensus metadata
I20260924 03:11:34.170368 16776 ts_tablet_manager.cc:1918] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:11:34.170372 16274 catalog_manager.cc:4841] TS 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): tablet a4dc231796c94ac7963dc2d2adc722f0 (table test-table [id=e22880e9fbbb4cc6a77a40899dba1aae]) successfully deleted
I20260924 03:11:34.170436 16776 log.cc:1196] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:34.170697 16776 ts_tablet_manager.cc:1939] T a4dc231796c94ac7963dc2d2adc722f0 P 3f74109c20ee460cbd9680363cdfbaf1: Deleting consensus metadata
I20260924 03:11:34.171175 16273 catalog_manager.cc:4841] TS 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): tablet a4dc231796c94ac7963dc2d2adc722f0 (table test-table [id=e22880e9fbbb4cc6a77a40899dba1aae]) successfully deleted
W20260924 03:11:34.172504 16272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37467 (error 108)
I20260924 03:11:34.172698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16480
I20260924 03:11:34.173214 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:34.172808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37467 (local address 127.0.0.1:42178)
0924 03:11:34.173145 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:34.173181 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
W20260924 03:11:34.173415 16272 catalog_manager.cc:4584] TS 07840626c95143278f76942e16f7a846 (127.0.98.1:37467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4dc231796c94ac7963dc2d2adc722f0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37467: connect: Connection refused (error 111)
I20260924 03:11:34.176952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16616
I20260924 03:11:34.181192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16254
I20260924 03:11:54.186060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--webserver_interface=127.0.98.62
--webserver_port=33889
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:54.260921 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:54.261163 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:54.261196 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:54.262287 16778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:11:54.262348 16778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:11:54.262367 16778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:11:54.262382 16778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:11:54.262406 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:54.262431 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:54.262452 16778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:11:54.262478 16778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:11:54.263744 16778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:54.263978 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:54.264187 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:11:54.268893 16778 server_base.cc:1034] running on GCE node
W20260924 03:11:54.269006 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.269073 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.269071 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:54.269373 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:54.269649 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:54.270808 16778 hybrid_clock.cc:648] HybridClock initialized: now 1790219514270815 us; error 39 us; skew 500 ppm
Sep 24 03:11:54 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:54.272975 16778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:11:54.273396 16778 webserver.cc:460] Webserver started at http://127.0.98.62:33889/ using document root <none> and password file <none>
I20260924 03:11:54.273584 16778 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:54.273623 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:54.274816 16778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:11:54.275372 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.275511 16778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:11:54.275596 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "3568def735cc4673950ed8415c531ce5"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "5503dfed6de81880c03361175ff9c883"
server_key_iv: "14bae06a05bbfa169cc8dac5e7d1ed9f"
server_key_version: "encryptionkey@0"
I20260924 03:11:54.275859 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:54.292695 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:54.294150 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:54.294329 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:54.297988 16778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46483
I20260924 03:11:54.298019 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46483 every 8 connection(s)
I20260924 03:11:54.298424 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:11:54.299717 16846 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:11:54.300253 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap starting.
I20260924 03:11:54.300698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16778
I20260924 03:11:54.300931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37467
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=44573
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:54.308810 16846 log.cc:824] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:54.310874 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:11:54.311141 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Bootstrap complete.
I20260924 03:11:54.313050 16846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:54.313247 16846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Initialized, Role: FOLLOWER
I20260924 03:11:54.313395 16846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:54.313469 16846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:11:54.313524 16846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:11:54.313582 16846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:11:54.314610 16846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:54.314692 16846 leader_election.cc:302] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3568def735cc4673950ed8415c531ce5; no voters:
I20260924 03:11:54.314872 16846 leader_election.cc:288] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:11:54.314903 16850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:11:54.315016 16850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [term 3 LEADER]: Becoming Leader. State: Replica: 3568def735cc4673950ed8415c531ce5, State: Running, Role: LEADER
I20260924 03:11:54.315116 16846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:11:54.315166 16850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260924 03:11:54.315420 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3568def735cc4673950ed8415c531ce5. Latest consensus state: current_term: 3 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:54.315423 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3568def735cc4673950ed8415c531ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3568def735cc4673950ed8415c531ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260924 03:11:54.315496 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:54.315521 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:11:54.315773 16856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:11:54.316530 16856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:11:54.316716 16856 catalog_manager.cc:1259] Loaded cluster ID: efdb5a97fe5f47ad93d2a1863215e03e
I20260924 03:11:54.316767 16856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:11:54.317997 16856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:11:54.318172 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Loaded TSK: 0
I20260924 03:11:54.318246 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3568def735cc4673950ed8415c531ce5: Loaded TSK: 1
F20260924 03:11:54.318470 16856 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff803bfb9ad google::LogMessage::Fail() at ??:0
@ 0x7ff803bfe88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff803bfb4de google::LogMessage::Flush() at ??:0
@ 0x7ff803bfca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff806368104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff806365d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff80bb551cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff80bb5548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff80bb55594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff80bb1bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff80bb25ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff805ac90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff805abe954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff8069226db start_thread at ??:0
@ 0x7ff8026f961f clone at ??:0
W20260924 03:11:54.393339 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:54.393527 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:54.393548 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:54.394655 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:54.394702 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:54.394726 16848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:11:54.396013 16848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:54.396219 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:54.396478 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:54.401052 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:54.401247 16848 server_base.cc:1034] running on GCE node
W20260924 03:11:54.401288 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.401304 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:54.401552 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:54.401782 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:54.402952 16848 hybrid_clock.cc:648] HybridClock initialized: now 1790219514402930 us; error 38 us; skew 500 ppm
Sep 24 03:11:54 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:54.405298 16848 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:11:54.405740 16848 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260924 03:11:54.405954 16848 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:54.405997 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:54.407307 16848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:54.408000 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.408174 16848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:54.408254 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "07840626c95143278f76942e16f7a846"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "563671654869596aca99a52619e66f5a"
server_key_iv: "6299c4d66e2b8d0db57a06d1c5db3806"
server_key_version: "encryptionkey@0"
I20260924 03:11:54.408584 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:54.417059 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:54.419157 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:54.419301 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:54.419497 16848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:54.419880 16904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:11:54.421752 16848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:11:54.421809 16848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:11:54.421851 16848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:11:54.422489 16848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:11:54.422541 16848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:11:54.422609 16904 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Bootstrap starting.
I20260924 03:11:54.429216 16848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37467
I20260924 03:11:54.429550 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:11:54.432881 16904 log.cc:824] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Log is configured to *not* fsync() on all Append() calls
I20260924 03:11:54.433136 17011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37467 every 8 connection(s)
I20260924 03:11:54.435307 16904 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:11:54.435598 16904 tablet_bootstrap.cc:492] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Bootstrap complete.
I20260924 03:11:54.435737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 16848
I20260924 03:11:54.435986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36421
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=37067
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:11:54.436761 16904 ts_tablet_manager.cc:1397] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260924 03:11:54.438169 16904 raft_consensus.cc:348] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:54.438508 16904 raft_consensus.cc:729] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07840626c95143278f76942e16f7a846, State: Initialized, Role: FOLLOWER
I20260924 03:11:54.438663 16904 consensus_queue.cc:260] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:54.438901 16904 ts_tablet_manager.cc:1428] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260924 03:11:54.498960 17014 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:11:54.430153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:40717)
0924 03:11:54.433084 (+ 2931us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.433154 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:54.433641 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:54.498801 (+ 65160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2892,"thread_start_us":162,"threads_started":1}
W20260924 03:11:54.499236 17012 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46483: Client connection negotiation failed: client connection to 127.0.98.62:46483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:11:54.499595 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.499485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:44497)
0924 03:11:54.499536 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.499579 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:11:54.499941 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.499847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:55423)
0924 03:11:54.499895 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.499918 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:11:54.500048 17012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:11:54.513882 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:54.514065 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:54.514107 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:54.515245 17016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:54.515309 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:54.515352 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:11:54.516772 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37067
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:54.517009 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:54.517225 17016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:11:54.521744 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.521754 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.521777 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:54.522017 17016 server_base.cc:1034] running on GCE node
I20260924 03:11:54.522251 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:54.522470 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:54.523612 17016 hybrid_clock.cc:648] HybridClock initialized: now 1790219514523598 us; error 28 us; skew 500 ppm
Sep 24 03:11:54 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:54.525714 17016 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:11:54.526166 17016 webserver.cc:460] Webserver started at http://127.0.98.2:37067/ using document root <none> and password file <none>
I20260924 03:11:54.526440 17016 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:54.526504 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:54.528146 17016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:11:54.528872 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.529088 17016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:11:54.529179 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "2d0f4a5c3d9742e38950a0400abd8e8e"
format_stamp: "Formatted at 2026-09-24 03:11:12 on dist-test-slave-jtnz"
server_key: "4c62310824d634302127d7cde17f249c"
server_key_iv: "ee8698357a26c20bb8b3b59c1c5915fc"
server_key_version: "encryptionkey@0"
I20260924 03:11:54.529605 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:54.557368 17016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:54.559032 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:54.559132 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:54.559289 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:54.559566 17016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:54.559613 17016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.559638 17016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:54.559686 17016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.565846 17016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36421
I20260924 03:11:54.565923 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36421 every 8 connection(s)
I20260924 03:11:54.566278 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:11:54.567137 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.566830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:36475)
0924 03:11:54.566963 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.567020 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
W20260924 03:11:54.567484 17146 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46483: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
I20260924 03:11:54.567705 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.567642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:55865)
0924 03:11:54.567667 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.567691 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260924 03:11:54.567952 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.567873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:53421)
0924 03:11:54.567922 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.567940 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 03:11:54.568048 17146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:11:54.571126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17016
I20260924 03:11:54.571411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37357
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46483
--webserver_port=33999
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:11:54.645502 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:11:54.645660 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:11:54.645679 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:11:54.646749 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:11:54.646787 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:11:54.646831 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:11:54.648175 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33999
--tserver_master_addrs=127.0.98.62:46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:11:54.648392 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:11:54.648628 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:11:54.653538 17149 server_base.cc:1034] running on GCE node
W20260924 03:11:54.653489 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.653489 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:11:54.653496 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:11:54.653927 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:11:54.654132 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:11:54.655279 17149 hybrid_clock.cc:648] HybridClock initialized: now 1790219514655249 us; error 25 us; skew 500 ppm
Sep 24 03:11:54 dist-test-slave-jtnz krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:11:54.657452 17149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:11:54.657848 17149 webserver.cc:460] Webserver started at http://127.0.98.3:33999/ using document root <none> and password file <none>
I20260924 03:11:54.658036 17149 fs_manager.cc:359] Metadata directory not provided
I20260924 03:11:54.658075 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:11:54.659320 17149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:54.660851 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.661057 17149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:11:54.661170 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "3f74109c20ee460cbd9680363cdfbaf1"
format_stamp: "Formatted at 2026-09-24 03:11:13 on dist-test-slave-jtnz"
server_key: "4fb33f8d7adde8efb15a0a58102d3c1b"
server_key_iv: "4c62277e33b50ed03ed01854f0e3729f"
server_key_version: "encryptionkey@0"
I20260924 03:11:54.661466 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:11:54.671105 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:11:54.672528 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:11:54.672665 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:11:54.672845 17149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:11:54.673081 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:11:54.673110 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.673128 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:11:54.673185 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:11:54.678764 17149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37357
I20260924 03:11:54.678813 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37357 every 8 connection(s)
I20260924 03:11:54.679132 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:11:54.680032 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.679733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:59237)
0924 03:11:54.679863 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.679919 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260924 03:11:54.680470 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46483: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
I20260924 03:11:54.680729 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.680643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:49675)
0924 03:11:54.680684 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.680710 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:11:54.681005 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:54.680933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:52783)
0924 03:11:54.680969 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:54.680990 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 03:11:54.681143 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:11:54.685768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17149
I20260924 03:11:55.500536 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.500341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:47989)
0924 03:11:55.500446 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:55.500477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260924 03:11:55.568535 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.568365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:59145)
0924 03:11:55.568460 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:55.568492 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260924 03:11:55.681564 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.681400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:41065)
0924 03:11:55.681495 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:55.681524 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260924 03:11:55.960502 17285 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:55.960610 17285 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:55.960858 17285 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
Sep 24 03:11:55 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 03:11:55 dist-test-slave-jtnz krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 03:11:55.970925 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.961010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37357 (local address 127.0.98.1:51255)
0924 03:11:55.961123 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:55.961133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:55.961183 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:55.961940 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:55.961941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:55.962373 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:55.962568 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:55.962574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.963711 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.963712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:55.964368 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:55.964370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.964942 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:55.965050 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:55.965604 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:55.965612 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:55.967445 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:55.969403 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:55.969404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:55.969644 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:55.969944 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:55.969945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:55.970000 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:55.970369 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:55.970376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:55.970516 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:11:55.970637 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:11:55.970804 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":62,"thread_start_us":67,"threads_started":1}
I20260924 03:11:55.970940 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.961200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51255 (local address 127.0.98.3:37357)
0924 03:11:55.961340 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:55.961347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:55.961369 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:11:55.961757 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:55.961759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:55.961792 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:55.961938 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:55.962686 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.963633 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964445 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.964852 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964967 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:55.967556 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:55.967559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:55.967559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:55.967567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:55.969332 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969793 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:55.969795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:55.969796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:55.969884 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:55.970059 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:55.970060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:55.970060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:55.970312 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:55.970380 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:55.970701 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 03:11:55.970818 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260924 03:11:55.970942 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.961120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32855 (local address 127.0.98.2:36421)
0924 03:11:55.961255 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 03:11:55.961267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:11:55.961302 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:11:55.961720 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:11:55.961722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:11:55.961762 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:11:55.961867 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:11:55.962727 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.963501 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964445 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.964854 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:55.967114 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:11:55.967117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:11:55.967117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:11:55.967139 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:11:55.969149 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969603 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:55.969605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:55.969606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:55.969765 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969994 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:11:55.969994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:11:55.969995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:11:55.970299 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:11:55.970380 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:11:55.970701 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 03:11:55.970818 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 03:11:55.970971 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:55.960996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36421 (local address 127.0.98.1:32855)
0924 03:11:55.961049 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:55.961061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:11:55.961128 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:11:55.961933 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:11:55.961936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:11:55.962370 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:11:55.962568 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:55.962574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.963578 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.963579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:55.964370 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:11:55.964372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:11:55.964944 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:11:55.964945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:11:55.965050 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:11:55.965672 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:11:55.965676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:11:55.967028 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:11:55.969227 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:55.969234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:55.969526 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:55.969824 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:11:55.969825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:11:55.969825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:11:55.969930 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:11:55.970369 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:11:55.970376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:11:55.970516 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:11:55.970640 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:11:55.970852 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":104}
W20260924 03:11:55.972108 16901 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
W20260924 03:11:55.972337 16900 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:55.972369 16900 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07840626c95143278f76942e16f7a846; no voters: 2d0f4a5c3d9742e38950a0400abd8e8e, 3f74109c20ee460cbd9680363cdfbaf1
I20260924 03:11:55.972474 17285 raft_consensus.cc:2726] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:56.501042 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:56.500856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:50877)
0924 03:11:56.500968 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:56.500992 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260924 03:11:56.569175 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:56.568989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:38125)
0924 03:11:56.569101 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:56.569134 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260924 03:11:56.682226 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:56.681976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:41261)
0924 03:11:56.682109 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:56.682151 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260924 03:11:57.501766 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:57.501548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:60685)
0924 03:11:57.501679 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:57.501711 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260924 03:11:57.512216 17293 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:57.512277 17293 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:57.512441 17293 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
W20260924 03:11:57.512784 16900 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
W20260924 03:11:57.512852 16901 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:57.512935 16901 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07840626c95143278f76942e16f7a846; no voters: 2d0f4a5c3d9742e38950a0400abd8e8e, 3f74109c20ee460cbd9680363cdfbaf1
I20260924 03:11:57.513003 17293 raft_consensus.cc:2726] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:57.569694 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:57.569515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:38669)
0924 03:11:57.569603 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:57.569633 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":41,"threads_started":1}
I20260924 03:11:57.682775 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:57.682614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:32957)
0924 03:11:57.682715 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:57.682745 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260924 03:11:58.502511 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:58.502315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:35325)
0924 03:11:58.502440 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:58.502469 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260924 03:11:58.570410 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:58.570040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:52537)
0924 03:11:58.570352 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:58.570380 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"threads_started":1}
I20260924 03:11:58.683337 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:58.683171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:46057)
0924 03:11:58.683277 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:58.683306 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260924 03:11:59.437203 17299 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:11:59.437276 17299 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:11:59.437490 17299 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
W20260924 03:11:59.437846 16900 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
W20260924 03:11:59.437935 16901 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:11:59.438016 16901 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07840626c95143278f76942e16f7a846; no voters: 2d0f4a5c3d9742e38950a0400abd8e8e, 3f74109c20ee460cbd9680363cdfbaf1
I20260924 03:11:59.438089 17299 raft_consensus.cc:2726] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:11:59.503120 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:59.502911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:40769)
0924 03:11:59.503046 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:59.503078 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260924 03:11:59.570927 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:59.570770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:53893)
0924 03:11:59.570870 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:59.570897 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260924 03:11:59.683880 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:11:59.683711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:59709)
0924 03:11:59.683815 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 03:11:59.683846 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260924 03:12:00.503841 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:00.503636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:36125)
0924 03:12:00.503767 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:00.503797 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 03:12:00.571410 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:00.571254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:43955)
0924 03:12:00.571351 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:00.571380 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260924 03:12:00.684696 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:00.684521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:37227)
0924 03:12:00.684636 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:00.684666 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260924 03:12:01.504575 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:01.504311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:59833)
0924 03:12:01.504499 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:01.504531 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":54,"threads_started":1}
I20260924 03:12:01.571936 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:01.571775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:41919)
0924 03:12:01.571876 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:01.571904 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 03:12:01.685497 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:01.685158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:45863)
0924 03:12:01.685430 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:01.685464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":69,"threads_started":1}
I20260924 03:12:01.739600 17309 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:01.739682 17309 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:12:01.739857 17309 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
W20260924 03:12:01.740231 16900 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
W20260924 03:12:01.740337 16901 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:12:01.740373 16901 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07840626c95143278f76942e16f7a846; no voters: 2d0f4a5c3d9742e38950a0400abd8e8e, 3f74109c20ee460cbd9680363cdfbaf1
I20260924 03:12:01.740469 17309 raft_consensus.cc:2726] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:02.505414 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:02.505211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:41761)
0924 03:12:02.505326 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:02.505359 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 03:12:02.572440 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:02.572284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:39761)
0924 03:12:02.572380 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:02.572410 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260924 03:12:02.686142 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:02.685970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:46667)
0924 03:12:02.686085 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:02.686111 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260924 03:12:03.506319 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:03.506065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:40029)
0924 03:12:03.506243 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:03.506274 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260924 03:12:03.572945 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:03.572781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:60169)
0924 03:12:03.572879 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:03.572918 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260924 03:12:03.686663 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:03.686500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:35265)
0924 03:12:03.686602 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:03.686631 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260924 03:12:04.506942 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:04.506757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:52509)
0924 03:12:04.506869 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:04.506899 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 03:12:04.573585 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:04.573419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:55561)
0924 03:12:04.573528 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:04.573557 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260924 03:12:04.687206 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:04.687039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:36881)
0924 03:12:04.687144 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:04.687175 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260924 03:12:05.507535 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:05.507332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:41807)
0924 03:12:05.507460 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:05.507492 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260924 03:12:05.574095 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:05.573914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:56105)
0924 03:12:05.574037 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:05.574063 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 03:12:05.687849 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:05.687565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:52865)
0924 03:12:05.687784 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:05.687816 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260924 03:12:06.109360 17322 raft_consensus.cc:482] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:06.109476 17322 raft_consensus.cc:504] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f74109c20ee460cbd9680363cdfbaf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37357 } } peers { permanent_uuid: "2d0f4a5c3d9742e38950a0400abd8e8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36421 } } peers { permanent_uuid: "07840626c95143278f76942e16f7a846" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37467 } }
I20260924 03:12:06.109675 17322 leader_election.cc:288] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357), 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421)
W20260924 03:12:06.110023 16901 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0f4a5c3d9742e38950a0400abd8e8e (127.0.98.2:36421): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
W20260924 03:12:06.110105 16900 leader_election.cc:341] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f74109c20ee460cbd9680363cdfbaf1 (127.0.98.3:37357): Not found: Tablet not found: a4dc231796c94ac7963dc2d2adc722f0
I20260924 03:12:06.110167 16900 leader_election.cc:302] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07840626c95143278f76942e16f7a846; no voters: 2d0f4a5c3d9742e38950a0400abd8e8e, 3f74109c20ee460cbd9680363cdfbaf1
I20260924 03:12:06.110262 17322 raft_consensus.cc:2726] T a4dc231796c94ac7963dc2d2adc722f0 P 07840626c95143278f76942e16f7a846 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:12:06.508237 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:06.508002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:42217)
0924 03:12:06.508165 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:06.508196 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260924 03:12:06.574683 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:06.574520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:51821)
0924 03:12:06.574616 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:06.574644 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260924 03:12:06.688383 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:06.688221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:37173)
0924 03:12:06.688317 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:06.688346 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260924 03:12:07.508927 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:07.508722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:38969)
0924 03:12:07.508849 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:07.508882 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260924 03:12:07.575189 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:07.575031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:47617)
0924 03:12:07.575129 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:07.575159 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260924 03:12:07.689252 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:07.689069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:41581)
0924 03:12:07.689192 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:07.689221 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260924 03:12:08.509670 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:08.509473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:40121)
0924 03:12:08.509593 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:08.509626 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 03:12:08.575764 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:08.575584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:36829)
0924 03:12:08.575704 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:08.575733 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260924 03:12:08.689815 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:08.689653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:43663)
0924 03:12:08.689751 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:08.689780 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260924 03:12:09.510627 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:09.510394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.1:40749)
0924 03:12:09.510528 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:09.510574 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260924 03:12:09.576297 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:09.576133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.2:52163)
0924 03:12:09.576238 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:09.576265 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260924 03:12:09.686789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 16848
I20260924 03:12:09.690526 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:09.690356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46483 (local address 127.0.98.3:51207)
0924 03:12:09.690461 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:09.690495 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260924 03:12:09.691169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17016
I20260924 03:12:09.694976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17149
2026-09-24T03:12:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58201 ms)
[----------] 30 tests from SecurityITest (219440 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:09 dist-test-slave-jtnz krb5kdc[17337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:09 dist-test-slave-jtnz krb5kdc[17337](info): set up 2 sockets
Sep 24 03:12:09 dist-test-slave-jtnz krb5kdc[17337](info): commencing operation
krb5kdc: starting...
W20260924 03:12:10.665752 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:10 dist-test-slave-jtnz krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:10.772881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:10.844304 17350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:10.844472 17350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:10.844493 17350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:10.845566 17350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:10.845611 17350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:10.845625 17350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:10.845638 17350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:10.845687 17350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:10.847041 17350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:10.847265 17350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:10.847477 17350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:10.852026 17350 server_base.cc:1034] running on GCE node
W20260924 03:12:10.852202 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:10.852155 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:10.852124 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:10.852630 17350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:10.852878 17350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:10.854020 17350 hybrid_clock.cc:648] HybridClock initialized: now 1790219530854006 us; error 35 us; skew 500 ppm
Sep 24 03:12:10 dist-test-slave-jtnz krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:10.856267 17350 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:10.856696 17350 webserver.cc:460] Webserver started at http://127.0.98.62:38305/ using document root <none> and password file <none>
I20260924 03:12:10.856894 17350 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:10.856953 17350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:10.857096 17350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:10.857995 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "5f019312ae0e4017b83f6b5d93e5db05"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-jtnz"
server_key: "926c3e17a0cad6c17468626734a8e4eb"
server_key_iv: "d94f82f03a6655a4a66ba108a590c5fc"
server_key_version: "encryptionkey@0"
I20260924 03:12:10.858367 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f019312ae0e4017b83f6b5d93e5db05"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-jtnz"
server_key: "926c3e17a0cad6c17468626734a8e4eb"
server_key_iv: "d94f82f03a6655a4a66ba108a590c5fc"
server_key_version: "encryptionkey@0"
I20260924 03:12:10.859578 17350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:12:10.860227 17365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:10.860383 17350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:10.860472 17350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "5f019312ae0e4017b83f6b5d93e5db05"
format_stamp: "Formatted at 2026-09-24 03:12:10 on dist-test-slave-jtnz"
server_key: "926c3e17a0cad6c17468626734a8e4eb"
server_key_iv: "d94f82f03a6655a4a66ba108a590c5fc"
server_key_version: "encryptionkey@0"
I20260924 03:12:10.860545 17350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:10.872583 17350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:10.874562 17350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:10.874699 17350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:10.878161 17350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37609
I20260924 03:12:10.878244 17417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37609 every 8 connection(s)
I20260924 03:12:10.878571 17350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:10.879308 17418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:10.881243 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: Bootstrap starting.
I20260924 03:12:10.881771 17418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:10.882076 17418 log.cc:824] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:10.882709 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: No bootstrap required, opened a new log
I20260924 03:12:10.884145 17418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } }
I20260924 03:12:10.884259 17418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:10.884279 17418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f019312ae0e4017b83f6b5d93e5db05, State: Initialized, Role: FOLLOWER
I20260924 03:12:10.884450 17418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } }
I20260924 03:12:10.884524 17418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:10.884572 17418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:10.884617 17418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:10.885071 17418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } }
I20260924 03:12:10.885172 17418 leader_election.cc:302] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f019312ae0e4017b83f6b5d93e5db05; no voters:
I20260924 03:12:10.885335 17418 leader_election.cc:288] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:10.885392 17421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:10.885547 17421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [term 1 LEADER]: Becoming Leader. State: Replica: 5f019312ae0e4017b83f6b5d93e5db05, State: Running, Role: LEADER
I20260924 03:12:10.885591 17418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:10.885637 17421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } }
I20260924 03:12:10.885986 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f019312ae0e4017b83f6b5d93e5db05. Latest consensus state: current_term: 1 leader_uuid: "5f019312ae0e4017b83f6b5d93e5db05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } } }
I20260924 03:12:10.886116 17422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f019312ae0e4017b83f6b5d93e5db05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f019312ae0e4017b83f6b5d93e5db05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37609 } } }
I20260924 03:12:10.886195 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:10.886287 17422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:10.886580 17430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:10.886756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17350
I20260924 03:12:10.886873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:10.887072 17430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:10.887082 392 external_mini_cluster.cc:1428] Setting key b846143d8ae0fceb5e42484d1e82cec1
I20260924 03:12:10.889053 17430 catalog_manager.cc:1347] Generated new cluster ID: e08432ec700643a8a58517719285332e
I20260924 03:12:10.889106 17430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:10 dist-test-slave-jtnz krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:10.896317 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:10.887523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.98.62:37609)
0924 03:12:10.887937 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:10.887944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:10.887994 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:12:10.888098 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:10.888104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:10.888168 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:10.888307 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:10.889016 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.889904 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.890744 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:10.891129 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:10.891248 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:10.893346 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:10.893349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:10.893349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:10.893371 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:10.894885 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:10.895237 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:10.895240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:10.895241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:10.895333 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:10.895535 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:10.895536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:10.895536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:10.895751 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:10.895907 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:10.895994 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 03:12:10.896154 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":104,"threads_started":1}
I20260924 03:12:10.899132 17430 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:10.899580 17430 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:10.906775 17430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: Generated new TSK 0
I20260924 03:12:10.906927 17430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:12:10.933051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:11.006031 17444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.006212 17444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.006234 17444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.007277 17444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.007313 17444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.007336 17444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:11.008569 17444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:11.008718 17444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.008858 17444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.013214 17450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.013191 17452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.013230 17444 server_base.cc:1034] running on GCE node
W20260924 03:12:11.013199 17449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.013616 17444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.013779 17444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.014899 17444 hybrid_clock.cc:648] HybridClock initialized: now 1790219531014887 us; error 25 us; skew 500 ppm
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:11.020654 17444 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:12:11.021023 17444 webserver.cc:460] Webserver started at http://127.0.98.1:43291/ using document root <none> and password file <none>
I20260924 03:12:11.021239 17444 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.021282 17444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.021425 17444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:11.022403 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "9738500c52c340a2a0997c1486762a9c"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "c1bcffe4cf6d34a27beaeb5913a46795"
server_key_iv: "d5bdaa76323320c3cdd8bdd8583164dd"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.022715 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9738500c52c340a2a0997c1486762a9c"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "c1bcffe4cf6d34a27beaeb5913a46795"
server_key_iv: "d5bdaa76323320c3cdd8bdd8583164dd"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.023835 17444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:11.024449 17459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.024610 17444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:11.024682 17444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "9738500c52c340a2a0997c1486762a9c"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "c1bcffe4cf6d34a27beaeb5913a46795"
server_key_iv: "d5bdaa76323320c3cdd8bdd8583164dd"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.024760 17444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.039093 17444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.040668 17444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.040807 17444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.041010 17444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:11.041285 17444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:11.041340 17444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.041378 17444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:11.041405 17444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.046586 17444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38345
I20260924 03:12:11.046622 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38345 every 8 connection(s)
I20260924 03:12:11.046885 17444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:11.055030 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.047488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48269 (local address 127.0.98.62:37609)
0924 03:12:11.047540 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.047542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.047693 (+ 151us) server_negotiation.cc:407] Connection header received
0924 03:12:11.048446 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.048447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.048461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.048524 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.049047 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.049521 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.050624 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.050839 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.050923 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.052908 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.052909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.052910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:11.052916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.054028 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.054392 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.054393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.054393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.054434 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.054614 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.054615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.054615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.054694 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.054734 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.054915 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.054973 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:12:11.055182 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.047466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37609 (local address 127.0.98.1:48269)
0924 03:12:11.047597 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.047624 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.048351 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.048582 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.048585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.048799 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:11.048950 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.048954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.049628 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.049629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.050535 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.050537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.050933 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.050934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.051048 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.051552 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:11.051560 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:11.052828 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:11.054091 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.054096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.054098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.054338 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.054483 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.054484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.054484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.054565 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.054736 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:11.054739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:11.054828 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.054902 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.054995 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260924 03:12:11.055719 17573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37609
I20260924 03:12:11.055785 17573 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:11.055895 17573 heartbeater.cc:511] Master 127.0.98.62:37609 requested a full tablet report, sending...
I20260924 03:12:11.056310 17382 ts_manager.cc:194] Registered new tserver with Master: 9738500c52c340a2a0997c1486762a9c (127.0.98.1:38345)
I20260924 03:12:11.056885 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48269
I20260924 03:12:11.057024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17444
I20260924 03:12:11.057124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:11.057307 392 external_mini_cluster.cc:1428] Setting key eb96d5cee5471e8851c0c173398e4dbf
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:12:11.078668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:11.164966 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.165191 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.165225 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.166319 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.166376 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.166415 17578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:11.167797 17578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:11.168044 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.168258 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.172838 17586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.172875 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.172845 17583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.173039 17578 server_base.cc:1034] running on GCE node
I20260924 03:12:11.173244 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.173447 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.174595 17578 hybrid_clock.cc:648] HybridClock initialized: now 1790219531174571 us; error 33 us; skew 500 ppm
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:11.176765 17578 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:12:11.177229 17578 webserver.cc:460] Webserver started at http://127.0.98.2:39947/ using document root <none> and password file <none>
I20260924 03:12:11.177435 17578 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.177505 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.177652 17578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:11.178679 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "76e41de33db6d0e54780706eb43669a9"
server_key_iv: "8740d6928dfa829025bd3733fe263d27"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.179028 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "76e41de33db6d0e54780706eb43669a9"
server_key_iv: "8740d6928dfa829025bd3733fe263d27"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.180236 17578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:11.180886 17593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.181056 17578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:11.181152 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "76e41de33db6d0e54780706eb43669a9"
server_key_iv: "8740d6928dfa829025bd3733fe263d27"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.181228 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.190868 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.192461 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.192596 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.192786 17578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:11.193035 17578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:11.193084 17578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.193127 17578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:11.193151 17578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.198524 17578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33319
I20260924 03:12:11.198562 17706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33319 every 8 connection(s)
I20260924 03:12:11.198822 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:11.202399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17578
I20260924 03:12:11.202502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:11.202692 392 external_mini_cluster.cc:1428] Setting key 5cce37c9179cfacf6daa5a449e1c4383
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:11.207764 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.199421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37609 (local address 127.0.98.2:57153)
0924 03:12:11.199569 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.199604 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.200471 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.200717 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.200720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.200969 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:11.201153 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.201157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.201824 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.201829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.202840 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.202843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.203234 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.203235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.203345 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.203789 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:11.203796 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:11.204961 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:11.206640 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.206645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.206647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.206830 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.207023 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.207024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.207024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.207108 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.207369 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:11.207372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:11.207462 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.207532 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.207635 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260924 03:12:11.207785 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.199447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57153 (local address 127.0.98.62:37609)
0924 03:12:11.199499 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.199501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.199673 (+ 172us) server_negotiation.cc:407] Connection header received
0924 03:12:11.200580 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.200581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.200593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.200655 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.201243 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.201734 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.202934 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.203133 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.203226 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.205066 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.205068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.205068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:11.205075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.206569 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.206901 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.206902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.206903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.206962 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.207168 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.207169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.207169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.207304 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.207377 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.207578 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.207662 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:12:11.208267 17707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37609
I20260924 03:12:11.208364 17707 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:11.208535 17707 heartbeater.cc:511] Master 127.0.98.62:37609 requested a full tablet report, sending...
I20260924 03:12:11.208887 17382 ts_manager.cc:194] Registered new tserver with Master: 1b6bbb41a0b64bbfab76ce6fb5e278c9 (127.0.98.2:33319)
I20260924 03:12:11.209368 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57153
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:12:11.226060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:11.299772 17712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.299937 17712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.299958 17712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.301095 17712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.301133 17712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.301184 17712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:11.302639 17712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:11.302860 17712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.303064 17712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.307466 17718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.307483 17717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.307483 17720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.307767 17712 server_base.cc:1034] running on GCE node
I20260924 03:12:11.307984 17712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.308200 17712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.309352 17712 hybrid_clock.cc:648] HybridClock initialized: now 1790219531309334 us; error 30 us; skew 500 ppm
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:11.311679 17712 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:12:11.312048 17712 webserver.cc:460] Webserver started at http://127.0.98.3:33533/ using document root <none> and password file <none>
I20260924 03:12:11.312249 17712 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.312310 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.312449 17712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:11.313340 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "82cd5c73745c417eb620565f2fd4d99d"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "fe92e3e7d83de32444931517990a599c"
server_key_iv: "9eca06622fd5fd52e5d6aabbd38c2d21"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.313668 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "82cd5c73745c417eb620565f2fd4d99d"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "fe92e3e7d83de32444931517990a599c"
server_key_iv: "9eca06622fd5fd52e5d6aabbd38c2d21"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.314982 17712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:11.315634 17727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.315786 17712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:11.315876 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "82cd5c73745c417eb620565f2fd4d99d"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "fe92e3e7d83de32444931517990a599c"
server_key_iv: "9eca06622fd5fd52e5d6aabbd38c2d21"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.315950 17712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.327675 17712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.329524 17712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.329671 17712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.329861 17712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:11.330135 17712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:11.330188 17712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.330265 17712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:11.330292 17712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.335745 17712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43287
I20260924 03:12:11.335803 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43287 every 8 connection(s)
I20260924 03:12:11.336084 17712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:11.339898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17712
I20260924 03:12:11.340005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:11.340179 392 external_mini_cluster.cc:1428] Setting key d4b8c9cdf217c90e6eb93f3db32073b6
Sep 24 03:12:11 dist-test-slave-jtnz krb5kdc[17337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:11.344543 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.336718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59581 (local address 127.0.98.62:37609)
0924 03:12:11.336792 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.336795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.336981 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:12:11.337807 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.337809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.337822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.337880 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.338547 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.339048 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.340040 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.340313 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.340378 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.342455 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.342456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.342456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:11.342463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.343521 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.343848 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.343849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.343849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.343897 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.344098 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.344098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.344099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.344192 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.344238 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.344430 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.344484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:12:11.344760 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.336676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37609 (local address 127.0.98.3:59581)
0924 03:12:11.336852 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.336900 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.337702 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.337939 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.337944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.338161 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:11.338444 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.338449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.339145 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.339146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.339942 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.339944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.340392 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.340393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.340532 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.341047 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:11.341055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:11.342374 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:11.343586 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.343591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.343593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.343786 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.343949 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:11.343950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:11.343950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:11.344034 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:11.344242 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:11.344245 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:11.344342 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.344415 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.344521 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260924 03:12:11.345232 17841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37609
I20260924 03:12:11.345319 17841 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:11.345494 17841 heartbeater.cc:511] Master 127.0.98.62:37609 requested a full tablet report, sending...
I20260924 03:12:11.345829 17382 ts_manager.cc:194] Registered new tserver with Master: 82cd5c73745c417eb620565f2fd4d99d (127.0.98.3:43287)
I20260924 03:12:11.346266 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59581
I20260924 03:12:11.351743 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:11.358321 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.353039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.98.62:37609)
0924 03:12:11.353113 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.353115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.353219 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:12:11.353263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.353264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.353271 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.353319 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.353850 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.354410 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.355074 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.355253 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.355299 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.356350 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.356351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.356352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:11.356357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.357404 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.357685 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.357686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.357687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.357732 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:11.357879 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:11.357879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:11.357879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:11.357997 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.358067 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.358146 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.358244 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:12:11.359674 17382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:11.361119 17382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:11.367468 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38345 (local address 127.0.0.1:41530)
0924 03:12:11.364164 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.364173 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.364230 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.364409 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.364410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.364413 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:11.364453 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.364455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365542 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.365543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.366475 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.366478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367161 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.367238 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367253 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.367348 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.367400 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":56,"threads_started":1}
I20260924 03:12:11.367484 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.1:38345)
0924 03:12:11.364115 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.364119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.364254 (+ 135us) server_negotiation.cc:407] Connection header received
0924 03:12:11.364302 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.364305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.364320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.364400 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:11.364517 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365429 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.366561 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367052 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367155 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367216 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.367366 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.367403 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260924 03:12:11.367707 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33319 (local address 127.0.0.1:38320)
0924 03:12:11.363583 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.363603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.363654 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.363989 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.363989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.363992 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:11.364046 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.364048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365107 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.365108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.366489 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.366492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367441 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.367512 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367516 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.367614 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.367641 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":35,"thread_start_us":50,"threads_started":1}
I20260924 03:12:11.367707 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43287 (local address 127.0.0.1:44958)
0924 03:12:11.363487 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.363529 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.363604 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.363891 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.363895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.363897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:11.363960 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.363962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365216 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.365216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.366490 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.366492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367443 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.367511 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367514 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.367564 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.367623 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":36,"thread_start_us":52,"threads_started":1}
I20260924 03:12:11.367861 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.98.3:43287)
0924 03:12:11.363534 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.363539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.363582 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:12:11.363681 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.363683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.363705 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.363806 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:11.364080 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365118 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.366599 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367286 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367355 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367403 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.367746 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.367788 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260924 03:12:11.367952 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.363627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.0.98.2:33319)
0924 03:12:11.363775 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.363780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.363790 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:11.363865 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.363867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.363883 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.364125 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:11.364213 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.365007 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.366591 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.367286 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.367355 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.367403 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.367823 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.367863 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260924 03:12:11.368687 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 4e50578277bf4569b81d771fc1e76d58 (DEFAULT_TABLE table=test-table [id=4451f35e0a284dc68c1caba04425ba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:11.368929 17775 tablet_service.cc:1467] Processing CreateTablet for tablet 4e50578277bf4569b81d771fc1e76d58 (DEFAULT_TABLE table=test-table [id=4451f35e0a284dc68c1caba04425ba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:11.369038 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e50578277bf4569b81d771fc1e76d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:11.369006 17507 tablet_service.cc:1467] Processing CreateTablet for tablet 4e50578277bf4569b81d771fc1e76d58 (DEFAULT_TABLE table=test-table [id=4451f35e0a284dc68c1caba04425ba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:11.369251 17775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e50578277bf4569b81d771fc1e76d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:11.369316 17507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e50578277bf4569b81d771fc1e76d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:11.371740 17855 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Bootstrap starting.
I20260924 03:12:11.372267 17856 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Bootstrap starting.
I20260924 03:12:11.372375 17855 tablet_bootstrap.cc:654] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:11.372637 17857 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Bootstrap starting.
I20260924 03:12:11.372995 17855 log.cc:824] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:11.373184 17856 tablet_bootstrap.cc:654] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:11.373265 17857 tablet_bootstrap.cc:654] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:11.373761 17857 log.cc:824] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:11.373806 17855 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: No bootstrap required, opened a new log
I20260924 03:12:11.373792 17856 log.cc:824] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:11.373862 17855 ts_tablet_manager.cc:1397] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:11.374459 17856 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: No bootstrap required, opened a new log
I20260924 03:12:11.374516 17857 tablet_bootstrap.cc:492] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: No bootstrap required, opened a new log
I20260924 03:12:11.374539 17856 ts_tablet_manager.cc:1397] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:11.374580 17857 ts_tablet_manager.cc:1397] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:11.375890 17855 raft_consensus.cc:348] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376065 17855 raft_consensus.cc:374] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:11.376101 17855 raft_consensus.cc:729] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cd5c73745c417eb620565f2fd4d99d, State: Initialized, Role: FOLLOWER
I20260924 03:12:11.376243 17855 consensus_queue.cc:260] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376349 17856 raft_consensus.cc:348] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376513 17855 ts_tablet_manager.cc:1428] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:11.376529 17856 raft_consensus.cc:374] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:11.376570 17856 raft_consensus.cc:729] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6bbb41a0b64bbfab76ce6fb5e278c9, State: Initialized, Role: FOLLOWER
I20260924 03:12:11.376631 17841 heartbeater.cc:503] Master 127.0.98.62:37609 was elected leader, sending a full tablet report...
I20260924 03:12:11.376526 17857 raft_consensus.cc:348] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376678 17857 raft_consensus.cc:374] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:11.376719 17857 raft_consensus.cc:729] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9738500c52c340a2a0997c1486762a9c, State: Initialized, Role: FOLLOWER
I20260924 03:12:11.376691 17856 consensus_queue.cc:260] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376829 17857 consensus_queue.cc:260] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.376963 17856 ts_tablet_manager.cc:1428] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:12:11.376998 17707 heartbeater.cc:503] Master 127.0.98.62:37609 was elected leader, sending a full tablet report...
I20260924 03:12:11.377110 17857 ts_tablet_manager.cc:1428] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:12:11.377220 17573 heartbeater.cc:503] Master 127.0.98.62:37609 was elected leader, sending a full tablet report...
W20260924 03:12:11.449344 17708 tablet.cc:2369] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:11.547477 17574 tablet.cc:2369] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:11.586668 17842 tablet.cc:2369] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:11.602926 17861 raft_consensus.cc:482] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:11.603039 17861 raft_consensus.cc:504] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.603392 17861 leader_election.cc:288] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9738500c52c340a2a0997c1486762a9c (127.0.98.1:38345), 1b6bbb41a0b64bbfab76ce6fb5e278c9 (127.0.98.2:33319)
I20260924 03:12:11.606097 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.603652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34231 (local address 127.0.98.1:38345)
0924 03:12:11.603711 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.603713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.603721 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:12:11.603750 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.603751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.603762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.603817 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:11.603985 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.604510 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605416 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.605753 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605802 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.605810 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.605980 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.606022 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:12:11.606089 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.603556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38345 (local address 127.0.98.3:34231)
0924 03:12:11.603607 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.603618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.603665 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.603834 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.603835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.603837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:11.603910 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.603912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.604594 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.604595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.605322 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.605324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.605828 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.605903 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.605906 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.605952 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.605981 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":17}
I20260924 03:12:11.606148 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.603620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54261 (local address 127.0.98.2:33319)
0924 03:12:11.603667 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.603668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.603839 (+ 171us) server_negotiation.cc:407] Connection header received
0924 03:12:11.603866 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.603867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.603877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.603925 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:11.604118 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.604537 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605451 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.605776 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605858 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.605866 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.606028 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.606078 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:12:11.606192 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.603573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33319 (local address 127.0.98.3:54261)
0924 03:12:11.603732 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.603741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.603803 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.603958 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.603959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.603961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:11.604031 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.604032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.604650 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.604651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.605337 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.605338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.605881 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.605881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.605935 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.605955 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.606008 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.606038 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260924 03:12:11.606535 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e50578277bf4569b81d771fc1e76d58" candidate_uuid: "82cd5c73745c417eb620565f2fd4d99d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9738500c52c340a2a0997c1486762a9c" is_pre_election: true
I20260924 03:12:11.606541 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e50578277bf4569b81d771fc1e76d58" candidate_uuid: "82cd5c73745c417eb620565f2fd4d99d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" is_pre_election: true
I20260924 03:12:11.606693 17527 raft_consensus.cc:2454] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cd5c73745c417eb620565f2fd4d99d in term 0.
I20260924 03:12:11.606701 17661 raft_consensus.cc:2454] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cd5c73745c417eb620565f2fd4d99d in term 0.
I20260924 03:12:11.606945 17728 leader_election.cc:302] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cd5c73745c417eb620565f2fd4d99d, 9738500c52c340a2a0997c1486762a9c; no voters:
I20260924 03:12:11.607129 17861 raft_consensus.cc:2781] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:11.607193 17861 raft_consensus.cc:482] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:11.607209 17861 raft_consensus.cc:3037] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:11.607910 17861 raft_consensus.cc:504] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.608047 17861 leader_election.cc:288] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [CANDIDATE]: Term 1 election: Requested vote from peers 9738500c52c340a2a0997c1486762a9c (127.0.98.1:38345), 1b6bbb41a0b64bbfab76ce6fb5e278c9 (127.0.98.2:33319)
I20260924 03:12:11.608210 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e50578277bf4569b81d771fc1e76d58" candidate_uuid: "82cd5c73745c417eb620565f2fd4d99d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9738500c52c340a2a0997c1486762a9c"
I20260924 03:12:11.608287 17527 raft_consensus.cc:3037] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:11.608994 17527 raft_consensus.cc:2454] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd5c73745c417eb620565f2fd4d99d in term 1.
I20260924 03:12:11.609154 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e50578277bf4569b81d771fc1e76d58" candidate_uuid: "82cd5c73745c417eb620565f2fd4d99d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9"
I20260924 03:12:11.609202 17728 leader_election.cc:302] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82cd5c73745c417eb620565f2fd4d99d, 9738500c52c340a2a0997c1486762a9c; no voters:
I20260924 03:12:11.609241 17661 raft_consensus.cc:3037] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:11.609342 17861 raft_consensus.cc:2781] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:11.609489 17861 raft_consensus.cc:686] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 1 LEADER]: Becoming Leader. State: Replica: 82cd5c73745c417eb620565f2fd4d99d, State: Running, Role: LEADER
I20260924 03:12:11.609581 17861 consensus_queue.cc:237] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } }
I20260924 03:12:11.610141 17661 raft_consensus.cc:2454] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cd5c73745c417eb620565f2fd4d99d in term 1.
I20260924 03:12:11.610667 17382 catalog_manager.cc:5495] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cd5c73745c417eb620565f2fd4d99d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "82cd5c73745c417eb620565f2fd4d99d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82cd5c73745c417eb620565f2fd4d99d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:11.650947 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.648699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.0.98.3:43287)
0924 03:12:11.648777 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.648779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.648788 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:11.648821 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.648826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.648840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.648899 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:11.649031 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.649586 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.650312 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.650490 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.650552 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.650679 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:11.650763 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:11.650822 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.650868 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.650883 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:12:11.652506 17661 raft_consensus.cc:1261] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 1 FOLLOWER]: Refusing update from remote peer 82cd5c73745c417eb620565f2fd4d99d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:11.652817 17527 raft_consensus.cc:1261] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 1 FOLLOWER]: Refusing update from remote peer 82cd5c73745c417eb620565f2fd4d99d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:11.652854 17865 consensus_queue.cc:1035] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6bbb41a0b64bbfab76ce6fb5e278c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:11.653055 17861 consensus_queue.cc:1035] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9738500c52c340a2a0997c1486762a9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:11.655349 17868 mvcc.cc:205] Tried to move back new op lower bound from 7332739201646166016 to 7332739201473445888. Current Snapshot: MvccSnapshot[applied={T|T < 7332739201646166016}]
I20260924 03:12:11.655431 17870 mvcc.cc:205] Tried to move back new op lower bound from 7332739201646166016 to 7332739201473445888. Current Snapshot: MvccSnapshot[applied={T|T < 7332739201646166016}]
I20260924 03:12:11.655581 17869 mvcc.cc:205] Tried to move back new op lower bound from 7332739201646166016 to 7332739201473445888. Current Snapshot: MvccSnapshot[applied={T|T < 7332739201646166016}]
I20260924 03:12:11.657596 17381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:11.657684 17381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:11.658422 17381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f019312ae0e4017b83f6b5d93e5db05: Sending DeleteTablet for 3 replicas of tablet 4e50578277bf4569b81d771fc1e76d58
I20260924 03:12:11.658751 17775 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e50578277bf4569b81d771fc1e76d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44958
I20260924 03:12:11.658752 17641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e50578277bf4569b81d771fc1e76d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38320
I20260924 03:12:11.658782 17507 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e50578277bf4569b81d771fc1e76d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41530
I20260924 03:12:11.658967 17879 tablet_replica.cc:330] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: stopping tablet replica
I20260924 03:12:11.658964 17880 tablet_replica.cc:330] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: stopping tablet replica
I20260924 03:12:11.659061 17879 raft_consensus.cc:2229] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:11.659061 17880 raft_consensus.cc:2229] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:11.659161 17879 raft_consensus.cc:2258] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:11.659162 17880 raft_consensus.cc:2258] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:11.659524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17444
I20260924 03:12:11.659574 17879 ts_tablet_manager.cc:1905] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:11.659574 17880 ts_tablet_manager.cc:1905] T 4e50578277bf4569b81d771fc1e76d58 P 9738500c52c340a2a0997c1486762a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:11.661454 17878 tablet_replica.cc:330] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: stopping tablet replica
I20260924 03:12:11.661538 17878 raft_consensus.cc:2229] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:11.661530 17879 ts_tablet_manager.cc:1918] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:11.661592 17879 log.cc:1196] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/4e50578277bf4569b81d771fc1e76d58
I20260924 03:12:11.661621 17878 raft_consensus.cc:2258] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:11.661944 17879 ts_tablet_manager.cc:1939] T 4e50578277bf4569b81d771fc1e76d58 P 82cd5c73745c417eb620565f2fd4d99d: Deleting consensus metadata
I20260924 03:12:11.661979 17878 ts_tablet_manager.cc:1905] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:11.663128 17366 catalog_manager.cc:4841] TS 82cd5c73745c417eb620565f2fd4d99d (127.0.98.3:43287): tablet 4e50578277bf4569b81d771fc1e76d58 (table test-table [id=4451f35e0a284dc68c1caba04425ba3f]) successfully deleted
I20260924 03:12:11.663839 17878 ts_tablet_manager.cc:1918] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:11.663897 17878 log.cc:1196] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/4e50578277bf4569b81d771fc1e76d58
I20260924 03:12:11.664143 17878 ts_tablet_manager.cc:1939] T 4e50578277bf4569b81d771fc1e76d58 P 1b6bbb41a0b64bbfab76ce6fb5e278c9: Deleting consensus metadata
I20260924 03:12:11.664640 17369 catalog_manager.cc:4841] TS 1b6bbb41a0b64bbfab76ce6fb5e278c9 (127.0.98.2:33319): tablet 4e50578277bf4569b81d771fc1e76d58 (table test-table [id=4451f35e0a284dc68c1caba04425ba3f]) successfully deleted
W20260924 03:12:11.666602 17366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38345 (error 108)
I20260924 03:12:11.667065 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.666930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38345 (local address 127.0.0.1:41538)
0924 03:12:11.666988 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.667035 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:12:11.667205 17366 catalog_manager.cc:4584] TS 9738500c52c340a2a0997c1486762a9c (127.0.98.1:38345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e50578277bf4569b81d771fc1e76d58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38345: connect: Connection refused (error 111)
I20260924 03:12:11.667295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17578
I20260924 03:12:11.672400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17712
I20260924 03:12:11.676443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17350
2026-09-24T03:12:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:11Z Disabled control of system clock
I20260924 03:12:11.702828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37101
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:11.776595 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.776762 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.776779 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.777765 17885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:11.777802 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.777813 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.777823 17885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:11.777869 17885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:11.779057 17885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:11.779286 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.779570 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.783897 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.783931 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.784049 17885 server_base.cc:1034] running on GCE node
W20260924 03:12:11.783893 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.784262 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.784499 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.785625 17885 hybrid_clock.cc:648] HybridClock initialized: now 1790219531785608 us; error 40 us; skew 500 ppm
I20260924 03:12:11.786801 17885 webserver.cc:460] Webserver started at http://127.0.98.62:44253/ using document root <none> and password file <none>
I20260924 03:12:11.787019 17885 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.787061 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.787197 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:11.788120 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "6a1739d174904305b6f13b87dbf5fad6"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ad478fd00a44e9252c3980bc5673aaa1"
server_key_iv: "c02172d6cb183115af9fe8fa9785ebae"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.788432 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a1739d174904305b6f13b87dbf5fad6"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ad478fd00a44e9252c3980bc5673aaa1"
server_key_iv: "c02172d6cb183115af9fe8fa9785ebae"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.789597 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:11.790213 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.790385 17885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:11.790460 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "6a1739d174904305b6f13b87dbf5fad6"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ad478fd00a44e9252c3980bc5673aaa1"
server_key_iv: "c02172d6cb183115af9fe8fa9785ebae"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.790509 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.805377 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.805568 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.805663 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.809371 17885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37101
I20260924 03:12:11.809397 17951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37101 every 8 connection(s)
I20260924 03:12:11.809696 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:11.810642 17952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:11.812528 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: Bootstrap starting.
I20260924 03:12:11.813056 17952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:11.813323 17952 log.cc:824] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:11.813913 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: No bootstrap required, opened a new log
I20260924 03:12:11.815402 17952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } }
I20260924 03:12:11.815565 17952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:11.815613 17952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1739d174904305b6f13b87dbf5fad6, State: Initialized, Role: FOLLOWER
I20260924 03:12:11.815749 17952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } }
I20260924 03:12:11.815838 17952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:11.815881 17952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:11.815935 17952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:11.816365 17952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } }
I20260924 03:12:11.816465 17952 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1739d174904305b6f13b87dbf5fad6; no voters:
I20260924 03:12:11.816571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17885
I20260924 03:12:11.816666 17952 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:11.816689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:11.816742 17955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:11.816887 392 external_mini_cluster.cc:1428] Setting key 876da5fa206ec30f0613aa967c59808b
I20260924 03:12:11.816883 17952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:11.816887 17955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1739d174904305b6f13b87dbf5fad6, State: Running, Role: LEADER
I20260924 03:12:11.817509 17955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } }
I20260924 03:12:11.818168 17957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1739d174904305b6f13b87dbf5fad6. Latest consensus state: current_term: 1 leader_uuid: "6a1739d174904305b6f13b87dbf5fad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } } }
I20260924 03:12:11.818558 17957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [sys.catalog]: This master's current role is: LEADER
W20260924 03:12:11.818626 17971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:11.818663 17971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:11.818169 17956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1739d174904305b6f13b87dbf5fad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1739d174904305b6f13b87dbf5fad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37101 } } }
I20260924 03:12:11.818745 17956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:11.818769 17973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:11.818928 17973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:11.820174 17973 catalog_manager.cc:1347] Generated new cluster ID: c866a66a144442509fe492856396cd80
I20260924 03:12:11.820225 17973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:11.823694 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.817287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.98.62:37101)
0924 03:12:11.817440 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.817448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.817567 (+ 119us) server_negotiation.cc:407] Connection header received
0924 03:12:11.817674 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.817680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.817741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.817874 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.819084 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.820325 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.821474 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.822056 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.822142 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.822971 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.822974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.822975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:11.823159 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.823193 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:11.823206 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.823245 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.823424 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.823499 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20260924 03:12:11.832381 17973 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:11.832793 17973 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:11.837033 17973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: Generated new TSK 0
I20260924 03:12:11.837186 17973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:11.838723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--builtin_ntp_servers=127.0.98.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:11.911173 17976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:11.911350 17976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:11.911370 17976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:11.912519 17976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:11.912552 17976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:11.912577 17976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:11.913789 17976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:11.913985 17976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:11.914289 17976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:11.918715 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.918706 17981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:11.918706 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:11.922389 17976 server_base.cc:1034] running on GCE node
I20260924 03:12:11.922600 17976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:11.922880 17976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:11.924062 17976 hybrid_clock.cc:648] HybridClock initialized: now 1790219531924045 us; error 40 us; skew 500 ppm
I20260924 03:12:11.925173 17976 webserver.cc:460] Webserver started at http://127.0.98.1:43581/ using document root <none> and password file <none>
I20260924 03:12:11.925379 17976 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:11.925422 17976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:11.925524 17976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:11.926643 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "015bf6111d8243cdb077edc15613acc5"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ebf315f8390af718803e892c6d4b036d"
server_key_iv: "be8a7e65bc85a26327dca9e59825240d"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.926962 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "015bf6111d8243cdb077edc15613acc5"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ebf315f8390af718803e892c6d4b036d"
server_key_iv: "be8a7e65bc85a26327dca9e59825240d"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.928227 17976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:11.928949 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.929141 17976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:11.929225 17976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "015bf6111d8243cdb077edc15613acc5"
format_stamp: "Formatted at 2026-09-24 03:12:11 on dist-test-slave-jtnz"
server_key: "ebf315f8390af718803e892c6d4b036d"
server_key_iv: "be8a7e65bc85a26327dca9e59825240d"
server_key_version: "encryptionkey@0"
I20260924 03:12:11.929291 17976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:11.937495 17976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:11.937678 17976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:11.937762 17976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:11.937896 17976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:11.938166 17976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:11.938251 17976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.938288 17976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:11.938308 17976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:11.943733 17976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34009
I20260924 03:12:11.943780 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34009 every 8 connection(s)
I20260924 03:12:11.944039 17976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:11.948396 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.944643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41237 (local address 127.0.98.62:37101)
0924 03:12:11.944711 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:11.944713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:11.944886 (+ 173us) server_negotiation.cc:407] Connection header received
0924 03:12:11.945270 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:11.945271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:11.945281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:11.945329 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:11.945903 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.946456 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.947349 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.947521 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.947572 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.948200 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:11.948201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:11.948201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:11.948205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:11.948216 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:11.948220 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:11.948263 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:11.948309 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 03:12:11.948330 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:12:11.948629 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:11.944606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37101 (local address 127.0.98.1:41237)
0924 03:12:11.944788 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:11.944815 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:11.945200 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:11.945372 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:11.945374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:11.945382 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:11.945840 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.945844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.946530 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.946530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.947278 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:11.947280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:11.947588 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:11.947588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:11.947684 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:11.948074 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:11.948074 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:11.948111 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:11.948130 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:11.948266 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:11.948268 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:11.948270 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:11.948311 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:12:11.948376 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260924 03:12:11.949162 18104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37101
I20260924 03:12:11.949236 18104 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:11.949398 18104 heartbeater.cc:511] Master 127.0.98.62:37101 requested a full tablet report, sending...
I20260924 03:12:11.949855 17916 ts_manager.cc:194] Registered new tserver with Master: 015bf6111d8243cdb077edc15613acc5 (127.0.98.1:34009)
I20260924 03:12:11.950469 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41237
I20260924 03:12:11.952847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 17976
I20260924 03:12:11.952917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:11.953029 392 external_mini_cluster.cc:1428] Setting key c1d93fd21320dd32aa14a30647612947
I20260924 03:12:11.953676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--builtin_ntp_servers=127.0.98.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:12.024937 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:12.025095 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:12.025113 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:12.026157 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:12.026212 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:12.026235 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:12.027400 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:12.027601 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:12.027854 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:12.032413 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:12.032513 18107 server_base.cc:1034] running on GCE node
W20260924 03:12:12.032400 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:12.032383 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:12.032840 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:12.033109 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:12.034235 18107 hybrid_clock.cc:648] HybridClock initialized: now 1790219532034219 us; error 30 us; skew 500 ppm
I20260924 03:12:12.035444 18107 webserver.cc:460] Webserver started at http://127.0.98.2:36457/ using document root <none> and password file <none>
I20260924 03:12:12.035627 18107 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:12.035665 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:12.035742 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:12.036603 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f343bd7157d410db752d8497cd707f2"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "1bdaea9332e3ee3d5d295cc22a5841f5"
server_key_iv: "dad55bf44aadcde4c8363bb753043365"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.036885 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f343bd7157d410db752d8497cd707f2"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "1bdaea9332e3ee3d5d295cc22a5841f5"
server_key_iv: "dad55bf44aadcde4c8363bb753043365"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.037994 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:12.038673 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.038815 18107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.038895 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "0f343bd7157d410db752d8497cd707f2"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "1bdaea9332e3ee3d5d295cc22a5841f5"
server_key_iv: "dad55bf44aadcde4c8363bb753043365"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.038962 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:12.051409 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:12.051604 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:12.051690 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:12.051838 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:12.052045 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:12.052071 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.052096 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:12.052115 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.057228 18107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39275
I20260924 03:12:12.057276 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39275 every 8 connection(s)
I20260924 03:12:12.057523 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:12.062000 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.058237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49463 (local address 127.0.98.62:37101)
0924 03:12:12.058308 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.058310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.058545 (+ 235us) server_negotiation.cc:407] Connection header received
0924 03:12:12.058931 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.058932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.058941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.058991 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.059536 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.059990 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.060881 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.061074 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.061123 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.061773 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.061774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.061774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:12.061778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.061789 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:12.061792 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.061834 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.061912 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.061933 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:12:12.062157 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.058161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37101 (local address 127.0.98.2:49463)
0924 03:12:12.058434 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.058462 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.058852 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.059031 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.059034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.059041 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:12.059472 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.059475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.060059 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.060059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.060808 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.060810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.061142 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.061143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.061239 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.061655 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:12.061655 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:12.061678 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:12.061697 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:12.061844 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:12.061847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:12.061849 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.061899 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.061980 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":180,"threads_started":1}
I20260924 03:12:12.062601 18235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37101
I20260924 03:12:12.062671 18235 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:12.062845 18235 heartbeater.cc:511] Master 127.0.98.62:37101 requested a full tablet report, sending...
I20260924 03:12:12.063205 17916 ts_manager.cc:194] Registered new tserver with Master: 0f343bd7157d410db752d8497cd707f2 (127.0.98.2:39275)
I20260924 03:12:12.063552 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49463
I20260924 03:12:12.067613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18107
I20260924 03:12:12.067692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:12.067873 392 external_mini_cluster.cc:1428] Setting key 31f0c0b918c9c417770376e800726bdf
I20260924 03:12:12.068704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--builtin_ntp_servers=127.0.98.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:12.141053 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:12.141216 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:12.141237 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:12.142401 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:12.142436 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:12.142457 18238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:12.143636 18238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:12.143849 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:12.144081 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:12.148250 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:12.148243 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:12.148470 18238 server_base.cc:1034] running on GCE node
W20260924 03:12:12.148267 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:12.148711 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:12.148924 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:12.150094 18238 hybrid_clock.cc:648] HybridClock initialized: now 1790219532150075 us; error 33 us; skew 500 ppm
I20260924 03:12:12.151240 18238 webserver.cc:460] Webserver started at http://127.0.98.3:38579/ using document root <none> and password file <none>
I20260924 03:12:12.151473 18238 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:12.151561 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:12.151685 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:12.152563 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4ddf50ebb954614b69f6b909ef364de"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "7d0c0cb51ffab4a13c516e08c8ccdfd7"
server_key_iv: "860a2ea91d654e9ef74d5814070636bd"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.152879 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ddf50ebb954614b69f6b909ef364de"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "7d0c0cb51ffab4a13c516e08c8ccdfd7"
server_key_iv: "860a2ea91d654e9ef74d5814070636bd"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.154094 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:12.154770 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.154929 18238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.155015 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "b4ddf50ebb954614b69f6b909ef364de"
format_stamp: "Formatted at 2026-09-24 03:12:12 on dist-test-slave-jtnz"
server_key: "7d0c0cb51ffab4a13c516e08c8ccdfd7"
server_key_iv: "860a2ea91d654e9ef74d5814070636bd"
server_key_version: "encryptionkey@0"
I20260924 03:12:12.155125 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:12.173952 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:12.174216 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:12.174342 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:12.174546 18238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:12.174815 18238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:12.174863 18238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.174890 18238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:12.174917 18238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:12.180301 18238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46475
I20260924 03:12:12.180372 18365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46475 every 8 connection(s)
I20260924 03:12:12.180675 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:12.182801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18238
I20260924 03:12:12.182910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:12.183079 392 external_mini_cluster.cc:1428] Setting key 5726269f35d09e8b167b4422e2e6f5fd
I20260924 03:12:12.185465 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.181371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45659 (local address 127.0.98.62:37101)
0924 03:12:12.181443 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.181445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.181627 (+ 182us) server_negotiation.cc:407] Connection header received
0924 03:12:12.182015 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.182016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.182025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.182074 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.182698 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.183274 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.184284 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.184472 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.184547 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.185176 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.185177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.185177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:12.185181 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.185193 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:12.185197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.185287 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.185370 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.185396 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:12:12.185679 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.181324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37101 (local address 127.0.98.3:45659)
0924 03:12:12.181508 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.181537 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.181942 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.182130 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.182133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.182140 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:12.182593 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.182597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.183358 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.183359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.184188 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.184190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.184563 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.184564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.184692 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.185063 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:12.185064 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:12.185084 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:12.185102 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:12.185290 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:12.185295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:12.185298 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.185373 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.185458 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260924 03:12:12.186151 18366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37101
I20260924 03:12:12.186278 18366 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:12.186426 18366 heartbeater.cc:511] Master 127.0.98.62:37101 requested a full tablet report, sending...
I20260924 03:12:12.186757 17916 ts_manager.cc:194] Registered new tserver with Master: b4ddf50ebb954614b69f6b909ef364de (127.0.98.3:46475)
I20260924 03:12:12.187069 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45659
I20260924 03:12:12.194677 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:12.198477 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.195808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.0.98.62:37101)
0924 03:12:12.195841 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.195843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.195990 (+ 147us) server_negotiation.cc:407] Connection header received
0924 03:12:12.196049 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.196050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.196060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.196103 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:12.196259 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.196667 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.197371 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.197540 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.197620 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.198252 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:12.198253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:12.198253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:12.198257 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:12.198268 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:12.198271 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:12.198319 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.198365 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.198386 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:12.199879 17916 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:12.201261 17916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:12.206836 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34009 (local address 127.0.0.1:37776)
0924 03:12:12.203394 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.203426 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.203481 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.203739 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.203740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.203742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:12.203783 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.203784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.204649 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.204650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.205782 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.205784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.206596 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.206596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.206666 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.206677 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.206732 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.206776 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":30,"thread_start_us":64,"threads_started":1}
I20260924 03:12:12.206925 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39275 (local address 127.0.0.1:47670)
0924 03:12:12.203403 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.203426 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.203481 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.203679 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.203682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.203684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:12.203735 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.203737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.204877 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.204878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.205846 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.205847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.206694 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.206695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.206753 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.206757 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.206804 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.206841 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260924 03:12:12.207126 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.0.98.1:34009)
0924 03:12:12.203550 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.203553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.203562 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:12.203605 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.203607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.203621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.203680 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:12.203841 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.204575 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.205886 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.206516 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.206593 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.206663 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.206995 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.207037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":78,"threads_started":1}
I20260924 03:12:12.207360 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.0.98.2:39275)
0924 03:12:12.203530 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.203533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.203542 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:12.203580 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.203581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.203595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.203670 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:12.203810 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.204811 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.205916 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.206538 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.206589 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.206663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.206818 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.206858 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260924 03:12:12.207741 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.98.3:46475)
0924 03:12:12.203467 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.203471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.203631 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:12:12.203716 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.203720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.203732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.203776 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:12.203919 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.204802 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.206742 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.207338 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.207423 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.207492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.207652 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.207683 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260924 03:12:12.207690 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 20f21c117ad44841972ea82832683140 (DEFAULT_TABLE table=test-table [id=e321d16ceac84a03922db15790342839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:12.207909 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f21c117ad44841972ea82832683140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:12.208030 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.203150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46475 (local address 127.0.0.1:39730)
0924 03:12:12.203561 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.203568 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.203630 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.203838 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.203839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.203840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:12.203868 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.203870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.205917 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.205917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.206672 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.206673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.207512 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.207513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.207567 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.207570 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.207939 (+ 369us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.207978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":82,"threads_started":1}
I20260924 03:12:12.207933 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 20f21c117ad44841972ea82832683140 (DEFAULT_TABLE table=test-table [id=e321d16ceac84a03922db15790342839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:12.208222 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f21c117ad44841972ea82832683140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:12.210100 18380 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: Bootstrap starting.
I20260924 03:12:12.210336 18381 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Bootstrap starting.
I20260924 03:12:12.210774 18380 tablet_bootstrap.cc:654] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:12.210950 18381 tablet_bootstrap.cc:654] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:12.210991 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 20f21c117ad44841972ea82832683140 (DEFAULT_TABLE table=test-table [id=e321d16ceac84a03922db15790342839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:12.211181 18380 log.cc:824] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:12.211269 18381 log.cc:824] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:12.211314 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f21c117ad44841972ea82832683140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:12.211881 18381 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: No bootstrap required, opened a new log
I20260924 03:12:12.211939 18380 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: No bootstrap required, opened a new log
I20260924 03:12:12.211959 18381 ts_tablet_manager.cc:1397] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:12.211997 18380 ts_tablet_manager.cc:1397] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:12.213515 18381 raft_consensus.cc:348] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.213657 18381 raft_consensus.cc:374] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:12.213697 18381 raft_consensus.cc:729] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f343bd7157d410db752d8497cd707f2, State: Initialized, Role: FOLLOWER
I20260924 03:12:12.213814 18381 consensus_queue.cc:260] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.214044 18381 ts_tablet_manager.cc:1428] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:12.214057 18235 heartbeater.cc:503] Master 127.0.98.62:37101 was elected leader, sending a full tablet report...
I20260924 03:12:12.214375 18385 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Bootstrap starting.
I20260924 03:12:12.214368 18380 raft_consensus.cc:348] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.214514 18380 raft_consensus.cc:374] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:12.214568 18380 raft_consensus.cc:729] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015bf6111d8243cdb077edc15613acc5, State: Initialized, Role: FOLLOWER
I20260924 03:12:12.214664 18380 consensus_queue.cc:260] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.214900 18380 ts_tablet_manager.cc:1428] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 03:12:12.214924 18385 tablet_bootstrap.cc:654] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:12.214931 18104 heartbeater.cc:503] Master 127.0.98.62:37101 was elected leader, sending a full tablet report...
I20260924 03:12:12.215194 18385 log.cc:824] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:12.215799 18385 tablet_bootstrap.cc:492] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: No bootstrap required, opened a new log
I20260924 03:12:12.215842 18385 ts_tablet_manager.cc:1397] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:12:12.217415 18385 raft_consensus.cc:348] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.217509 18385 raft_consensus.cc:374] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:12.217526 18385 raft_consensus.cc:729] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ddf50ebb954614b69f6b909ef364de, State: Initialized, Role: FOLLOWER
I20260924 03:12:12.217644 18385 consensus_queue.cc:260] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.217860 18385 ts_tablet_manager.cc:1428] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:12:12.217903 18366 heartbeater.cc:503] Master 127.0.98.62:37101 was elected leader, sending a full tablet report...
W20260924 03:12:12.308022 18236 tablet.cc:2369] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:12.322148 18384 raft_consensus.cc:482] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:12.322264 18384 raft_consensus.cc:504] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.322830 18384 leader_election.cc:288] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 015bf6111d8243cdb077edc15613acc5 (127.0.98.1:34009), b4ddf50ebb954614b69f6b909ef364de (127.0.98.3:46475)
I20260924 03:12:12.325489 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.323036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40533 (local address 127.0.98.3:46475)
0924 03:12:12.323092 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.323094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.323196 (+ 102us) server_negotiation.cc:407] Connection header received
0924 03:12:12.323227 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.323228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.323237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.323300 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:12.323476 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.323915 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.324822 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.325176 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.325225 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.325232 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.325382 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.325417 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:12:12.325500 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.322971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46475 (local address 127.0.98.2:40533)
0924 03:12:12.323129 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.323138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.323185 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.323342 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.323342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.323344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:12.323407 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.323409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.324003 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.324004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.324739 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.324741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.325247 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.325248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.325317 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.325320 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.325368 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.325409 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260924 03:12:12.325584 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.323012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57953 (local address 127.0.98.1:34009)
0924 03:12:12.323053 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.323054 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.323062 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:12:12.323087 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.323088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.323099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.323160 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:12.323348 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.323856 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.324889 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.325252 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.325316 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.325325 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.325479 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.325517 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:12:12.325767 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.322929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34009 (local address 127.0.98.2:57953)
0924 03:12:12.322976 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.322988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:12.323025 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:12.323195 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:12.323197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:12.323199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:12.323270 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.323272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.323935 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.323936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.324792 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:12.324795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.325334 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.325335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:12.325407 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.325410 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:12.325461 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:12:12.325494 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":19}
I20260924 03:12:12.325874 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f21c117ad44841972ea82832683140" candidate_uuid: "0f343bd7157d410db752d8497cd707f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ddf50ebb954614b69f6b909ef364de" is_pre_election: true
I20260924 03:12:12.326038 18320 raft_consensus.cc:2454] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f343bd7157d410db752d8497cd707f2 in term 0.
I20260924 03:12:12.326068 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f21c117ad44841972ea82832683140" candidate_uuid: "0f343bd7157d410db752d8497cd707f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015bf6111d8243cdb077edc15613acc5" is_pre_election: true
I20260924 03:12:12.326210 18058 raft_consensus.cc:2454] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f343bd7157d410db752d8497cd707f2 in term 0.
I20260924 03:12:12.326290 18123 leader_election.cc:302] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f343bd7157d410db752d8497cd707f2, b4ddf50ebb954614b69f6b909ef364de; no voters:
I20260924 03:12:12.326413 18384 raft_consensus.cc:2781] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:12.326467 18384 raft_consensus.cc:482] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:12.326503 18384 raft_consensus.cc:3037] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:12.327128 18384 raft_consensus.cc:504] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.327252 18384 leader_election.cc:288] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [CANDIDATE]: Term 1 election: Requested vote from peers 015bf6111d8243cdb077edc15613acc5 (127.0.98.1:34009), b4ddf50ebb954614b69f6b909ef364de (127.0.98.3:46475)
I20260924 03:12:12.327371 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f21c117ad44841972ea82832683140" candidate_uuid: "0f343bd7157d410db752d8497cd707f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015bf6111d8243cdb077edc15613acc5"
I20260924 03:12:12.327392 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f21c117ad44841972ea82832683140" candidate_uuid: "0f343bd7157d410db752d8497cd707f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ddf50ebb954614b69f6b909ef364de"
I20260924 03:12:12.327458 18058 raft_consensus.cc:3037] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:12.327494 18320 raft_consensus.cc:3037] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:12.328172 18320 raft_consensus.cc:2454] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f343bd7157d410db752d8497cd707f2 in term 1.
I20260924 03:12:12.328231 18058 raft_consensus.cc:2454] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f343bd7157d410db752d8497cd707f2 in term 1.
I20260924 03:12:12.328368 18123 leader_election.cc:302] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f343bd7157d410db752d8497cd707f2, b4ddf50ebb954614b69f6b909ef364de; no voters:
I20260924 03:12:12.328444 18384 raft_consensus.cc:2781] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:12.328596 18384 raft_consensus.cc:686] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 1 LEADER]: Becoming Leader. State: Replica: 0f343bd7157d410db752d8497cd707f2, State: Running, Role: LEADER
I20260924 03:12:12.328701 18384 consensus_queue.cc:237] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } }
I20260924 03:12:12.329391 17916 catalog_manager.cc:5495] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f343bd7157d410db752d8497cd707f2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f343bd7157d410db752d8497cd707f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f343bd7157d410db752d8497cd707f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:12.349704 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.347461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.0.98.2:39275)
0924 03:12:12.347498 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:12.347499 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:12.347508 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:12.347558 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:12.347559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:12.347580 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:12.347658 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:12.347779 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.348221 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.348966 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:12.349139 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:12.349209 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:12.349388 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:12.349477 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:12.349582 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:12.349601 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:12:12.349618 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:12.351300 18320 raft_consensus.cc:1261] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 1 FOLLOWER]: Refusing update from remote peer 0f343bd7157d410db752d8497cd707f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:12.351320 18058 raft_consensus.cc:1261] T 20f21c117ad44841972ea82832683140 P 015bf6111d8243cdb077edc15613acc5 [term 1 FOLLOWER]: Refusing update from remote peer 0f343bd7157d410db752d8497cd707f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:12.351656 18390 consensus_queue.cc:1035] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4ddf50ebb954614b69f6b909ef364de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:12.351771 18384 consensus_queue.cc:1035] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015bf6111d8243cdb077edc15613acc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:12.353654 18393 mvcc.cc:205] Tried to move back new op lower bound from 7332739204508499968 to 7332739204418863104. Current Snapshot: MvccSnapshot[applied={T|T < 7332739204508499968}]
I20260924 03:12:12.354421 18394 mvcc.cc:205] Tried to move back new op lower bound from 7332739204508499968 to 7332739204418863104. Current Snapshot: MvccSnapshot[applied={T|T < 7332739204508499968 or (T in {7332739204508499968})}]
I20260924 03:12:12.354579 18395 mvcc.cc:205] Tried to move back new op lower bound from 7332739204508499968 to 7332739204418863104. Current Snapshot: MvccSnapshot[applied={T|T < 7332739204508499968 or (T in {7332739204508499968})}]
I20260924 03:12:12.356390 17916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:12.356467 17916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:12.357079 17916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a1739d174904305b6f13b87dbf5fad6: Sending DeleteTablet for 3 replicas of tablet 20f21c117ad44841972ea82832683140
I20260924 03:12:12.357369 18169 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f21c117ad44841972ea82832683140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:12 UTC) from {username='slave'} at 127.0.0.1:47670
I20260924 03:12:12.357372 18300 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f21c117ad44841972ea82832683140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:12 UTC) from {username='slave'} at 127.0.0.1:39730
I20260924 03:12:12.357407 18038 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f21c117ad44841972ea82832683140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:12 UTC) from {username='slave'} at 127.0.0.1:37776
I20260924 03:12:12.357584 18402 tablet_replica.cc:330] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: stopping tablet replica
I20260924 03:12:12.357668 18402 raft_consensus.cc:2229] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:12.357669 18403 tablet_replica.cc:330] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: stopping tablet replica
I20260924 03:12:12.357746 18403 raft_consensus.cc:2229] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:12.357780 18402 raft_consensus.cc:2258] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:12.357828 18403 raft_consensus.cc:2258] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:12.358016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17976
I20260924 03:12:12.359505 18402 ts_tablet_manager.cc:1905] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:12.360744 18403 ts_tablet_manager.cc:1905] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:12.361588 18402 ts_tablet_manager.cc:1918] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:12.361693 18402 log.cc:1196] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/20f21c117ad44841972ea82832683140
I20260924 03:12:12.361994 18402 ts_tablet_manager.cc:1939] T 20f21c117ad44841972ea82832683140 P 0f343bd7157d410db752d8497cd707f2: Deleting consensus metadata
I20260924 03:12:12.362303 18403 ts_tablet_manager.cc:1918] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:12.362368 18403 log.cc:1196] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/20f21c117ad44841972ea82832683140
I20260924 03:12:12.362416 17902 catalog_manager.cc:4841] TS 0f343bd7157d410db752d8497cd707f2 (127.0.98.2:39275): tablet 20f21c117ad44841972ea82832683140 (table test-table [id=e321d16ceac84a03922db15790342839]) successfully deleted
I20260924 03:12:12.362610 18403 ts_tablet_manager.cc:1939] T 20f21c117ad44841972ea82832683140 P b4ddf50ebb954614b69f6b909ef364de: Deleting consensus metadata
I20260924 03:12:12.362977 17901 catalog_manager.cc:4841] TS b4ddf50ebb954614b69f6b909ef364de (127.0.98.3:46475): tablet 20f21c117ad44841972ea82832683140 (table test-table [id=e321d16ceac84a03922db15790342839]) successfully deleted
W20260924 03:12:12.364439 17903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34009 (error 108)
I20260924 03:12:12.364632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18107
I20260924 03:12:12.365348 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:12.364739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34009 (local address 127.0.0.1:37780)
0924 03:12:12.365288 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:12.365325 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532}
W20260924 03:12:12.365463 17903 catalog_manager.cc:4584] TS 015bf6111d8243cdb077edc15613acc5 (127.0.98.1:34009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20f21c117ad44841972ea82832683140: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34009: connect: Connection refused (error 111)
I20260924 03:12:12.369181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18238
I20260924 03:12:12.373680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 17885
2026-09-24T03:12:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:12 dist-test-slave-jtnz krb5kdc[18408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:12 dist-test-slave-jtnz krb5kdc[18408](info): set up 2 sockets
Sep 24 03:12:12 dist-test-slave-jtnz krb5kdc[18408](info): commencing operation
krb5kdc: starting...
W20260924 03:12:13.345490 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:13.444816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:13.518217 18421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.518391 18421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.518411 18421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.519508 18421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:13.519548 18421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.519562 18421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.519639 18421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:13.519659 18421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:13.521104 18421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:13.521338 18421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.521549 18421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:13.525899 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.525913 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.525962 18421 server_base.cc:1034] running on GCE node
W20260924 03:12:13.525913 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.526391 18421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.526643 18421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.527772 18421 hybrid_clock.cc:648] HybridClock initialized: now 1790219533527755 us; error 34 us; skew 500 ppm
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.529896 18421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:13.530339 18421 webserver.cc:460] Webserver started at http://127.0.98.62:46417/ using document root <none> and password file <none>
I20260924 03:12:13.530573 18421 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.530638 18421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:13.530769 18421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:13.531730 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "b06efe845ee845149566848221496cac"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "a0f016d905938153e53cf99ceef3d37e"
server_key_iv: "9c19360f47acfff2378c8a0c60724196"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.532066 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "b06efe845ee845149566848221496cac"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "a0f016d905938153e53cf99ceef3d37e"
server_key_iv: "9c19360f47acfff2378c8a0c60724196"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.533356 18421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:13.533983 18436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.534138 18421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:13.534245 18421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "b06efe845ee845149566848221496cac"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "a0f016d905938153e53cf99ceef3d37e"
server_key_iv: "9c19360f47acfff2378c8a0c60724196"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.534320 18421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:13.545703 18421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:13.547322 18421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:13.547466 18421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:13.551105 18421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45663
I20260924 03:12:13.551136 18488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45663 every 8 connection(s)
I20260924 03:12:13.551437 18421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:13.552186 18489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:13.554095 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: Bootstrap starting.
I20260924 03:12:13.554607 18489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:13.554858 18489 log.cc:824] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:13.555439 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: No bootstrap required, opened a new log
I20260924 03:12:13.556815 18489 raft_consensus.cc:348] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 03:12:13.556939 18489 raft_consensus.cc:374] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:13.556978 18489 raft_consensus.cc:729] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06efe845ee845149566848221496cac, State: Initialized, Role: FOLLOWER
I20260924 03:12:13.557122 18489 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 03:12:13.557199 18489 raft_consensus.cc:388] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:13.557245 18489 raft_consensus.cc:482] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:13.557289 18489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:13.557788 18489 raft_consensus.cc:504] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 03:12:13.557890 18489 leader_election.cc:302] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06efe845ee845149566848221496cac; no voters:
I20260924 03:12:13.558072 18489 leader_election.cc:288] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:13.558097 18492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:13.558166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18421
I20260924 03:12:13.558231 18492 raft_consensus.cc:686] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [term 1 LEADER]: Becoming Leader. State: Replica: b06efe845ee845149566848221496cac, State: Running, Role: LEADER
I20260924 03:12:13.558295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:13.558300 18492 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } }
I20260924 03:12:13.558333 18489 sys_catalog.cc:564] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:13.558470 392 external_mini_cluster.cc:1428] Setting key 8ada3cf32fb9ab79cf16d3b6c4d9f954
I20260924 03:12:13.558580 18492 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06efe845ee845149566848221496cac. Latest consensus state: current_term: 1 leader_uuid: "b06efe845ee845149566848221496cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 03:12:13.558781 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06efe845ee845149566848221496cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06efe845ee845149566848221496cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45663 } } }
I20260924 03:12:13.558868 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:13.558768 18492 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:13.559228 18501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:13.559722 18501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:13.561218 18501 catalog_manager.cc:1347] Generated new cluster ID: 041d54b6924d4acda36dd1b4bf4d94bf
I20260924 03:12:13.561265 18501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:13.569793 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:13.559089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.98.62:45663)
0924 03:12:13.559278 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:13.559286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:13.559314 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:12:13.559473 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:13.559480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:13.559531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:13.559654 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:13.560573 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.561877 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.562945 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.563484 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.563570 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:13.565968 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:13.565972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:13.565972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:13.565982 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:13.568329 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.568687 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.568690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.568690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.568809 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.568997 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.568998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.568998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.569278 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:13.569433 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:13.569492 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:12:13.569610 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260924 03:12:13.577426 18501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:13.577853 18501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:13.581277 18501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: Generated new TSK 0
I20260924 03:12:13.581423 18501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:12:13.606936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:13.684587 18515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.684751 18515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.684769 18515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.685953 18515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.685993 18515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.686019 18515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:13.687484 18515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:13.687731 18515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.687964 18515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:13.692760 18515 server_base.cc:1034] running on GCE node
W20260924 03:12:13.692718 18520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.692741 18521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.692744 18523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.693166 18515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.693396 18515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.694550 18515 hybrid_clock.cc:648] HybridClock initialized: now 1790219533694535 us; error 27 us; skew 500 ppm
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.696777 18515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:12:13.697180 18515 webserver.cc:460] Webserver started at http://127.0.98.1:33385/ using document root <none> and password file <none>
I20260924 03:12:13.697386 18515 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.697446 18515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:13.697582 18515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:13.698582 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5c4766d64f743eb9b6d37c75d7ae24d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "18c2684afc635303a854c33f3ba747ed"
server_key_iv: "1360711fb23edde115077afc33ab7163"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.698904 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5c4766d64f743eb9b6d37c75d7ae24d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "18c2684afc635303a854c33f3ba747ed"
server_key_iv: "1360711fb23edde115077afc33ab7163"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.700054 18515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:13.700740 18530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.700901 18515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:13.701011 18515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "f5c4766d64f743eb9b6d37c75d7ae24d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "18c2684afc635303a854c33f3ba747ed"
server_key_iv: "1360711fb23edde115077afc33ab7163"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.701097 18515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:13.711124 18515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:13.712788 18515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:13.712947 18515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:13.713133 18515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:13.713411 18515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:13.713460 18515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.713500 18515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:13.713537 18515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.718788 18515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43641
I20260924 03:12:13.718837 18643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43641 every 8 connection(s)
I20260924 03:12:13.719099 18515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:13.720992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18515
I20260924 03:12:13.721103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:13.721285 392 external_mini_cluster.cc:1428] Setting key 32e84260d6497929827ee915118d6dc7
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:13.728291 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:13.719670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54747 (local address 127.0.98.62:45663)
0924 03:12:13.719715 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:13.719717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:13.719935 (+ 218us) server_negotiation.cc:407] Connection header received
0924 03:12:13.720794 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:13.720795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:13.720823 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:13.720892 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:13.721595 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.722338 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.723260 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.723495 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.723549 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:13.725700 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:13.725701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:13.725702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:13.725708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:13.726990 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.727399 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.727400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.727400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.727442 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.727704 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.727705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.727705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.727788 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:13.727832 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:13.728166 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 03:12:13.728231 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:13.728533 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:13.719648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.1:54747)
0924 03:12:13.719795 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:13.719826 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:13.720696 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:13.720969 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:13.720972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:13.721270 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:13.721503 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:13.721508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.722413 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.722414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:13.723170 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:13.723173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.723572 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.723579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:13.723713 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:13.724150 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:13.724158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:13.725613 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:13.727060 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:13.727067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:13.727070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:13.727334 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:13.727499 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:13.727500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:13.727500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:13.727640 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:13.727848 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:13.727853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:13.728021 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 03:12:13.728144 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:12:13.728297 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 03:12:13.729079 18644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 03:12:13.729161 18644 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:12:13.729333 18644 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 03:12:13.729794 18453 ts_manager.cc:194] Registered new tserver with Master: f5c4766d64f743eb9b6d37c75d7ae24d (127.0.98.1:43641)
I20260924 03:12:13.730376 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:12:13.744488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:13.819119 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.819307 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.819340 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.820343 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.820405 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.820446 18649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:13.821882 18649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:13.822134 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.822340 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:13.826673 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.826721 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.826668 18654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.826808 18649 server_base.cc:1034] running on GCE node
I20260924 03:12:13.826965 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.827178 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.828332 18649 hybrid_clock.cc:648] HybridClock initialized: now 1790219533828314 us; error 30 us; skew 500 ppm
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.830667 18649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:12:13.831048 18649 webserver.cc:460] Webserver started at http://127.0.98.2:36407/ using document root <none> and password file <none>
I20260924 03:12:13.831251 18649 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.831315 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:13.831457 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:13.832374 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "1436107c3f3f41d4ad5c50856f1cc432"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "fbe5cef841ae192dcbcd7b2fdfcf3e7f"
server_key_iv: "e9ed9473bd6f7b6caf090142d03a703f"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.832705 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1436107c3f3f41d4ad5c50856f1cc432"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "fbe5cef841ae192dcbcd7b2fdfcf3e7f"
server_key_iv: "e9ed9473bd6f7b6caf090142d03a703f"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.833945 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:12:13.834616 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.834754 18649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.834838 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "1436107c3f3f41d4ad5c50856f1cc432"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "fbe5cef841ae192dcbcd7b2fdfcf3e7f"
server_key_iv: "e9ed9473bd6f7b6caf090142d03a703f"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.834909 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:13.844256 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:13.846230 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:13.846364 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:13.846592 18649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:13.846859 18649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:13.846905 18649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.846941 18649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:13.846966 18649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.852423 18649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46457
I20260924 03:12:13.852471 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46457 every 8 connection(s)
I20260924 03:12:13.852736 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:13.858232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18649
I20260924 03:12:13.858341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:13.858598 392 external_mini_cluster.cc:1428] Setting key d1cfe4d26b843307e1e75105f5e51455
I20260924 03:12:13.862246 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:13.853352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52171 (local address 127.0.98.62:45663)
0924 03:12:13.853414 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:13.853416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:13.853602 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:12:13.854516 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:13.854517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:13.854530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:13.854597 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:13.855148 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.855629 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.856584 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.856789 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.856869 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:13.859066 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:13.859067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:13.859068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:13.859074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:13.860786 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.861297 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.861298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.861298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.861350 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:13.861619 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:13.861620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:13.861620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:13.861715 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:13.861764 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:13.862076 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:12:13.862136 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:12:13.862522 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:13.853345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.2:52171)
0924 03:12:13.853485 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:13.853519 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:13.854401 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:13.854659 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:13.854663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:13.854886 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:13.855044 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:13.855047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.855733 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.855734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:13.856485 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:13.856487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:13.856879 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:13.856880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:13.856995 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:13.857367 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:13.857374 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:13.858981 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:13.860874 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:13.860882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:13.860884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:13.861215 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:13.861418 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:13.861419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:13.861419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:13.861554 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:13.861781 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:13.861786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:13.861943 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 03:12:13.862062 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:12:13.862256 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 03:12:13.862939 18778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 03:12:13.863039 18778 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:13.863217 18778 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 03:12:13.863564 18453 ts_manager.cc:194] Registered new tserver with Master: 1436107c3f3f41d4ad5c50856f1cc432 (127.0.98.2:46457)
I20260924 03:12:13.863925 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52171
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:12:13.881891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:13.955084 18783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:13.955243 18783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:13.955263 18783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:13.956221 18783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:13.956256 18783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:13.956301 18783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:13.957684 18783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:13.957924 18783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:13.958078 18783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:13.962653 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.962707 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:13.962651 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:13.962813 18783 server_base.cc:1034] running on GCE node
I20260924 03:12:13.963047 18783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:13.963213 18783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:13.964345 18783 hybrid_clock.cc:648] HybridClock initialized: now 1790219533964337 us; error 24 us; skew 500 ppm
Sep 24 03:12:13 dist-test-slave-jtnz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:13.966735 18783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:12:13.967118 18783 webserver.cc:460] Webserver started at http://127.0.98.3:38215/ using document root <none> and password file <none>
I20260924 03:12:13.967320 18783 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:13.967362 18783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:13.967478 18783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:13.968341 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ba259b47e704111a763aea84c6a420d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "dd5dd576362283cc63370de4784c045c"
server_key_iv: "165cd4e0f301ad9fb3ab101a721c0d90"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.968649 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ba259b47e704111a763aea84c6a420d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "dd5dd576362283cc63370de4784c045c"
server_key_iv: "165cd4e0f301ad9fb3ab101a721c0d90"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.969779 18783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:12:13.970464 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:13.970715 18783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:12:13.970783 18783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "1ba259b47e704111a763aea84c6a420d"
format_stamp: "Formatted at 2026-09-24 03:12:13 on dist-test-slave-jtnz"
server_key: "dd5dd576362283cc63370de4784c045c"
server_key_iv: "165cd4e0f301ad9fb3ab101a721c0d90"
server_key_version: "encryptionkey@0"
I20260924 03:12:13.970870 18783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:13.997195 18783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:13.999337 18783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:13.999459 18783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:13.999689 18783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:13.999966 18783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:13.999996 18783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.000048 18783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.000077 18783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.005764 18783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34743
I20260924 03:12:14.005818 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34743 every 8 connection(s)
I20260924 03:12:14.006084 18783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:12:14 dist-test-slave-jtnz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:14.013859 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.006722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42389 (local address 127.0.98.62:45663)
0924 03:12:14.006762 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.006764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.006943 (+ 179us) server_negotiation.cc:407] Connection header received
0924 03:12:14.007747 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.007748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.007757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.007807 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.008292 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.008722 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.009574 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.009721 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.009762 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.011670 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.011671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.011672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:14.011678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.012848 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.013181 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.013182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.013182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.013228 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.013430 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.013431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.013431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.013515 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.013558 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.013749 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.013802 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:14.014016 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.006693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45663 (local address 127.0.98.3:42389)
0924 03:12:14.006846 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.006883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.007668 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.007855 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.007858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.008064 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.008229 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.008233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.008786 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.008787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.009514 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.009516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.009780 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.009781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.009866 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.010262 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:14.010274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.011588 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.012914 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.012919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.012921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.013120 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.013282 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:14.013282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:14.013283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:14.013383 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:14.013562 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.013565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.013655 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.013724 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.013819 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260924 03:12:14.014533 18912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45663
I20260924 03:12:14.014603 18912 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.014765 18912 heartbeater.cc:511] Master 127.0.98.62:45663 requested a full tablet report, sending...
I20260924 03:12:14.015103 18453 ts_manager.cc:194] Registered new tserver with Master: 1ba259b47e704111a763aea84c6a420d (127.0.98.3:34743)
I20260924 03:12:14.015390 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42389
I20260924 03:12:14.015920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18783
I20260924 03:12:14.015995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:14.016173 392 external_mini_cluster.cc:1428] Setting key f777ff5c1c08a9e6491d27ce52662e76
I20260924 03:12:14.016942 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:14.023346 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.018114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.0.98.62:45663)
0924 03:12:14.018164 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.018166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.018287 (+ 121us) server_negotiation.cc:407] Connection header received
0924 03:12:14.018331 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.018332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.018339 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.018377 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.018867 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.019288 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.019931 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.020091 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.020135 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.021366 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.021367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.021367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:14.021371 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.022470 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.022776 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.022777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.022777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.022820 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:14.022969 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:14.022969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:14.022969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:14.023087 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.023155 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.023256 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.023297 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:12:14.024622 18453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:14.026002 18453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:14.031858 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.027865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34743 (local address 127.0.0.1:38850)
0924 03:12:14.028004 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.028047 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.028101 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.028466 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.028467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.028468 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.028497 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.028498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029768 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.029769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.030785 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.030789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.031672 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.031733 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.031742 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.031786 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.031815 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":105,"thread_start_us":43,"threads_started":1}
I20260924 03:12:14.031915 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.028141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.0.98.3:34743)
0924 03:12:14.028325 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.028328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.028338 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:14.028366 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.028368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.028392 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.028466 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.028565 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029684 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.030899 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.031568 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031655 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.031717 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.031796 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.031829 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260924 03:12:14.032316 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.027859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46457 (local address 127.0.0.1:35664)
0924 03:12:14.027994 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.028034 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.028102 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.028435 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.028437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.028439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.028496 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.028498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029877 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.029878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.031027 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.031029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.031920 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.032130 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.032134 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.032193 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.032245 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":35,"thread_start_us":89,"threads_started":1}
I20260924 03:12:14.032335 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.027950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.0.98.2:46457)
0924 03:12:14.028227 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.028231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.028240 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:14.028294 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.028296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.028312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.028425 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.028552 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029682 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031109 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.031837 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031901 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.031949 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.032206 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.032261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":90,"threads_started":1}
I20260924 03:12:14.032676 18846 tablet_service.cc:1467] Processing CreateTablet for tablet b08e07059bde46bfa1e1f16520f1b084 (DEFAULT_TABLE table=test-table [id=048426b1ec584cb9bd13ba80e3b5284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.032970 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.027886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.0.1:57866)
0924 03:12:14.028256 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.028281 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.028353 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.028723 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.028724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.028726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.028817 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.028819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029719 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.029720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.031011 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.031015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.032775 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.032776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.032832 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.032836 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.032877 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.032912 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":63,"threads_started":1}
I20260924 03:12:14.033005 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08e07059bde46bfa1e1f16520f1b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.033164 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.028100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.0.98.1:43641)
0924 03:12:14.028237 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.028241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.028353 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:12:14.028610 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.028613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.028627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.028727 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.028895 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.029610 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.031985 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.032697 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.032773 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.032820 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.033059 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.033097 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260924 03:12:14.033231 18712 tablet_service.cc:1467] Processing CreateTablet for tablet b08e07059bde46bfa1e1f16520f1b084 (DEFAULT_TABLE table=test-table [id=048426b1ec584cb9bd13ba80e3b5284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.033506 18712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08e07059bde46bfa1e1f16520f1b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.033746 18578 tablet_service.cc:1467] Processing CreateTablet for tablet b08e07059bde46bfa1e1f16520f1b084 (DEFAULT_TABLE table=test-table [id=048426b1ec584cb9bd13ba80e3b5284c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.034107 18578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08e07059bde46bfa1e1f16520f1b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.035622 18926 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Bootstrap starting.
I20260924 03:12:14.036204 18926 tablet_bootstrap.cc:654] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.036530 18927 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Bootstrap starting.
I20260924 03:12:14.036684 18926 log.cc:824] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.037129 18927 tablet_bootstrap.cc:654] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.037222 18929 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: Bootstrap starting.
I20260924 03:12:14.037479 18926 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: No bootstrap required, opened a new log
I20260924 03:12:14.037561 18926 ts_tablet_manager.cc:1397] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.037688 18927 log.cc:824] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.037894 18929 tablet_bootstrap.cc:654] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.038426 18929 log.cc:824] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.038480 18927 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: No bootstrap required, opened a new log
I20260924 03:12:14.038532 18927 ts_tablet_manager.cc:1397] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.039157 18929 tablet_bootstrap.cc:492] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: No bootstrap required, opened a new log
I20260924 03:12:14.039238 18929 ts_tablet_manager.cc:1397] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.039381 18926 raft_consensus.cc:348] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.039556 18926 raft_consensus.cc:374] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.039613 18926 raft_consensus.cc:729] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba259b47e704111a763aea84c6a420d, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.039744 18926 consensus_queue.cc:260] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.039997 18926 ts_tablet_manager.cc:1428] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:14.040001 18912 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 03:12:14.040484 18927 raft_consensus.cc:348] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.040663 18927 raft_consensus.cc:374] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.040699 18927 raft_consensus.cc:729] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1436107c3f3f41d4ad5c50856f1cc432, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.040829 18927 consensus_queue.cc:260] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.041116 18927 ts_tablet_manager.cc:1428] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:12:14.041232 18778 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 03:12:14.041232 18929 raft_consensus.cc:348] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.041380 18929 raft_consensus.cc:374] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.041425 18929 raft_consensus.cc:729] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c4766d64f743eb9b6d37c75d7ae24d, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.041553 18929 consensus_queue.cc:260] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.041824 18644 heartbeater.cc:503] Master 127.0.98.62:45663 was elected leader, sending a full tablet report...
I20260924 03:12:14.041826 18929 ts_tablet_manager.cc:1428] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 03:12:14.103272 18779 tablet.cc:2369] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:14.161247 18933 raft_consensus.cc:482] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:14.161350 18933 raft_consensus.cc:504] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.161772 18933 leader_election.cc:288] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ba259b47e704111a763aea84c6a420d (127.0.98.3:34743), f5c4766d64f743eb9b6d37c75d7ae24d (127.0.98.1:43641)
I20260924 03:12:14.165463 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.162004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48655 (local address 127.0.98.3:34743)
0924 03:12:14.162049 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.162050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.162199 (+ 149us) server_negotiation.cc:407] Connection header received
0924 03:12:14.162246 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.162247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.162258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.162308 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.162435 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.163160 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.164576 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.165107 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.165172 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.165182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.165360 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.165395 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:12:14.165463 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.162032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56827 (local address 127.0.98.1:43641)
0924 03:12:14.162078 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.162080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.162090 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:14.162173 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.162174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.162186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.162270 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.162430 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.163153 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.164580 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.165101 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.165173 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.165182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.165359 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.165395 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:12:14.165483 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.98.2:56827)
0924 03:12:14.162022 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.162042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.162090 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.162272 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.162273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.162274 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.162345 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.162346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.163239 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.163239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.164488 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.164491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.165189 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.165190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.165274 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.165278 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.165335 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.165379 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":79}
I20260924 03:12:14.165489 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.161914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34743 (local address 127.0.98.2:48655)
0924 03:12:14.162112 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.162121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.162171 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.162309 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.162310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.162311 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.162367 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.162368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.163239 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.163239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.164488 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.164491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.165189 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.165190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.165274 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.165278 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.165335 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.165379 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":70,"thread_start_us":104,"threads_started":1}
I20260924 03:12:14.165771 18598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08e07059bde46bfa1e1f16520f1b084" candidate_uuid: "1436107c3f3f41d4ad5c50856f1cc432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" is_pre_election: true
I20260924 03:12:14.165904 18598 raft_consensus.cc:2454] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1436107c3f3f41d4ad5c50856f1cc432 in term 0.
I20260924 03:12:14.166043 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08e07059bde46bfa1e1f16520f1b084" candidate_uuid: "1436107c3f3f41d4ad5c50856f1cc432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba259b47e704111a763aea84c6a420d" is_pre_election: true
I20260924 03:12:14.166110 18666 leader_election.cc:302] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1436107c3f3f41d4ad5c50856f1cc432, f5c4766d64f743eb9b6d37c75d7ae24d; no voters:
I20260924 03:12:14.166222 18866 raft_consensus.cc:2454] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1436107c3f3f41d4ad5c50856f1cc432 in term 0.
I20260924 03:12:14.166245 18933 raft_consensus.cc:2781] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:14.166311 18933 raft_consensus.cc:482] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:14.166340 18933 raft_consensus.cc:3037] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.167004 18933 raft_consensus.cc:504] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.167133 18933 leader_election.cc:288] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [CANDIDATE]: Term 1 election: Requested vote from peers 1ba259b47e704111a763aea84c6a420d (127.0.98.3:34743), f5c4766d64f743eb9b6d37c75d7ae24d (127.0.98.1:43641)
I20260924 03:12:14.167271 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08e07059bde46bfa1e1f16520f1b084" candidate_uuid: "1436107c3f3f41d4ad5c50856f1cc432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba259b47e704111a763aea84c6a420d"
I20260924 03:12:14.167284 18598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08e07059bde46bfa1e1f16520f1b084" candidate_uuid: "1436107c3f3f41d4ad5c50856f1cc432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d"
I20260924 03:12:14.167378 18598 raft_consensus.cc:3037] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.167378 18866 raft_consensus.cc:3037] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.168044 18866 raft_consensus.cc:2454] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1436107c3f3f41d4ad5c50856f1cc432 in term 1.
I20260924 03:12:14.168116 18598 raft_consensus.cc:2454] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1436107c3f3f41d4ad5c50856f1cc432 in term 1.
I20260924 03:12:14.168220 18667 leader_election.cc:302] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1436107c3f3f41d4ad5c50856f1cc432, 1ba259b47e704111a763aea84c6a420d; no voters:
I20260924 03:12:14.168311 18933 raft_consensus.cc:2781] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:14.168471 18933 raft_consensus.cc:686] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 1 LEADER]: Becoming Leader. State: Replica: 1436107c3f3f41d4ad5c50856f1cc432, State: Running, Role: LEADER
I20260924 03:12:14.168567 18933 consensus_queue.cc:237] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260924 03:12:14.169276 18453 catalog_manager.cc:5495] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1436107c3f3f41d4ad5c50856f1cc432 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1436107c3f3f41d4ad5c50856f1cc432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1436107c3f3f41d4ad5c50856f1cc432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:14.178548 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.176280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.0.98.2:46457)
0924 03:12:14.176316 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.176318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.176361 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:12:14.176447 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.176448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.176468 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.176537 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:14.176651 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.177052 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.177727 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.177894 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.177978 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.178206 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:14.178300 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:14.178409 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.178456 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.178466 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:12:14.180226 18598 raft_consensus.cc:1261] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 1 FOLLOWER]: Refusing update from remote peer 1436107c3f3f41d4ad5c50856f1cc432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:14.180305 18866 raft_consensus.cc:1261] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 1 FOLLOWER]: Refusing update from remote peer 1436107c3f3f41d4ad5c50856f1cc432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:14.180506 18936 consensus_queue.cc:1035] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5c4766d64f743eb9b6d37c75d7ae24d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:14.180697 18933 consensus_queue.cc:1035] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba259b47e704111a763aea84c6a420d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:14.182896 18939 mvcc.cc:205] Tried to move back new op lower bound from 7332739211999543296 to 7332739211954855936. Current Snapshot: MvccSnapshot[applied={T|T < 7332739211999543296}]
I20260924 03:12:14.183451 18941 mvcc.cc:205] Tried to move back new op lower bound from 7332739211999543296 to 7332739211954855936. Current Snapshot: MvccSnapshot[applied={T|T < 7332739211999543296 or (T in {7332739211999543296})}]
I20260924 03:12:14.183866 18940 mvcc.cc:205] Tried to move back new op lower bound from 7332739211999543296 to 7332739211954855936. Current Snapshot: MvccSnapshot[applied={T|T < 7332739211999543296}]
I20260924 03:12:14.185542 18452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:14.185626 18452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:14.186352 18452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b06efe845ee845149566848221496cac: Sending DeleteTablet for 3 replicas of tablet b08e07059bde46bfa1e1f16520f1b084
I20260924 03:12:14.186650 18712 tablet_service.cc:1514] Processing DeleteTablet for tablet b08e07059bde46bfa1e1f16520f1b084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35664
I20260924 03:12:14.186674 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet b08e07059bde46bfa1e1f16520f1b084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38850
I20260924 03:12:14.186898 18951 tablet_replica.cc:330] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: stopping tablet replica
I20260924 03:12:14.186896 18950 tablet_replica.cc:330] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: stopping tablet replica
I20260924 03:12:14.186890 18578 tablet_service.cc:1514] Processing DeleteTablet for tablet b08e07059bde46bfa1e1f16520f1b084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57866
I20260924 03:12:14.186986 18950 raft_consensus.cc:2229] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:14.186986 18951 raft_consensus.cc:2229] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:14.187089 18951 raft_consensus.cc:2258] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:14.187112 18950 raft_consensus.cc:2258] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:14.187112 18952 tablet_replica.cc:330] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d: stopping tablet replica
I20260924 03:12:14.187192 18952 raft_consensus.cc:2229] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:14.187270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18515
I20260924 03:12:14.187315 18952 raft_consensus.cc:2258] T b08e07059bde46bfa1e1f16520f1b084 P f5c4766d64f743eb9b6d37c75d7ae24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:14.188072 18950 ts_tablet_manager.cc:1905] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:14.188689 18951 ts_tablet_manager.cc:1905] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:14.189776 18950 ts_tablet_manager.cc:1918] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:14.189842 18950 log.cc:1196] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/b08e07059bde46bfa1e1f16520f1b084
I20260924 03:12:14.190114 18950 ts_tablet_manager.cc:1939] T b08e07059bde46bfa1e1f16520f1b084 P 1436107c3f3f41d4ad5c50856f1cc432: Deleting consensus metadata
I20260924 03:12:14.190315 18951 ts_tablet_manager.cc:1918] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:14.190382 18951 log.cc:1196] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/b08e07059bde46bfa1e1f16520f1b084
I20260924 03:12:14.190552 18437 catalog_manager.cc:4841] TS 1436107c3f3f41d4ad5c50856f1cc432 (127.0.98.2:46457): tablet b08e07059bde46bfa1e1f16520f1b084 (table test-table [id=048426b1ec584cb9bd13ba80e3b5284c]) successfully deleted
I20260924 03:12:14.190673 18951 ts_tablet_manager.cc:1939] T b08e07059bde46bfa1e1f16520f1b084 P 1ba259b47e704111a763aea84c6a420d: Deleting consensus metadata
I20260924 03:12:14.191020 18439 catalog_manager.cc:4841] TS 1ba259b47e704111a763aea84c6a420d (127.0.98.3:34743): tablet b08e07059bde46bfa1e1f16520f1b084 (table test-table [id=048426b1ec584cb9bd13ba80e3b5284c]) successfully deleted
W20260924 03:12:14.192991 18438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43641 (error 108)
I20260924 03:12:14.193208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18649
I20260924 03:12:14.193394 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.193291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.0.1:57872)
0924 03:12:14.193333 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.193361 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 03:12:14.193943 18438 catalog_manager.cc:4584] TS f5c4766d64f743eb9b6d37c75d7ae24d (127.0.98.1:43641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b08e07059bde46bfa1e1f16520f1b084: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43641: connect: Connection refused (error 111)
I20260924 03:12:14.197721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18783
I20260924 03:12:14.201778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18421
2026-09-24T03:12:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:14Z Disabled control of system clock
I20260924 03:12:14.227622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:14.302289 18957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:14.302462 18957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:14.302481 18957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:14.303491 18957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:14.303520 18957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:14.303534 18957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:14.303545 18957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:14.303556 18957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:14.304726 18957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:14.304883 18957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:14.305083 18957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:14.309393 18965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.309540 18962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.309489 18957 server_base.cc:1034] running on GCE node
W20260924 03:12:14.309413 18963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.309808 18957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:14.310111 18957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:14.311268 18957 hybrid_clock.cc:648] HybridClock initialized: now 1790219534311260 us; error 51 us; skew 500 ppm
I20260924 03:12:14.312354 18957 webserver.cc:460] Webserver started at http://127.0.98.62:41113/ using document root <none> and password file <none>
I20260924 03:12:14.312559 18957 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:14.312621 18957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:14.312738 18957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:14.313683 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "f5a53d9bfc1748b28e288719526567df"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "0dcf112bdba876ee64e4603e17afa3ba"
server_key_iv: "ee6ec75d6a3d3f17d30af7e1cd23cecf"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.314014 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5a53d9bfc1748b28e288719526567df"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "0dcf112bdba876ee64e4603e17afa3ba"
server_key_iv: "ee6ec75d6a3d3f17d30af7e1cd23cecf"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.315230 18957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:12:14.315855 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.316020 18957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:14.316107 18957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "f5a53d9bfc1748b28e288719526567df"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "0dcf112bdba876ee64e4603e17afa3ba"
server_key_iv: "ee6ec75d6a3d3f17d30af7e1cd23cecf"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.316182 18957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.353366 18957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:14.353655 18957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.353801 18957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.357612 18957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44405
I20260924 03:12:14.357642 19023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44405 every 8 connection(s)
I20260924 03:12:14.357936 18957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:14.358865 19024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.360791 19024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: Bootstrap starting.
I20260924 03:12:14.361286 19024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.361512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 18957
I20260924 03:12:14.361511 19024 log.cc:824] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.361619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:14.361805 392 external_mini_cluster.cc:1428] Setting key 27e53b01f1825cc44ece4a143d858990
I20260924 03:12:14.362250 19024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: No bootstrap required, opened a new log
I20260924 03:12:14.363821 19024 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260924 03:12:14.363965 19024 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.364038 19024 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a53d9bfc1748b28e288719526567df, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.364193 19024 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260924 03:12:14.364284 19024 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:14.364316 19024 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:14.364362 19024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.364856 19024 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260924 03:12:14.364934 19024 leader_election.cc:302] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a53d9bfc1748b28e288719526567df; no voters:
I20260924 03:12:14.365111 19024 leader_election.cc:288] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:14.365141 19029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:14.365253 19029 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [term 1 LEADER]: Becoming Leader. State: Replica: f5a53d9bfc1748b28e288719526567df, State: Running, Role: LEADER
I20260924 03:12:14.365358 19024 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:14.365377 19029 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } }
I20260924 03:12:14.365864 19030 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a53d9bfc1748b28e288719526567df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } }
I20260924 03:12:14.365955 19030 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:14.366194 19031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a53d9bfc1748b28e288719526567df. Latest consensus state: current_term: 1 leader_uuid: "f5a53d9bfc1748b28e288719526567df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a53d9bfc1748b28e288719526567df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } }
I20260924 03:12:14.366258 19031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:14.366432 19036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:14.366933 19036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:14.368644 19036 catalog_manager.cc:1347] Generated new cluster ID: a1611fd7110d49459f094481eef19238
I20260924 03:12:14.368695 19036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:14.369117 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.362262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.0.98.62:44405)
0924 03:12:14.362453 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.362460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.362482 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:14.363808 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.363814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.363866 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.364004 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.364255 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.365878 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.367099 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.367714 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.367846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.368455 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.368458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.368459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:14.368634 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.368676 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:14.368691 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.368754 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.368870 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.368941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260924 03:12:14.388489 19036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:14.389074 19036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:14.396448 19036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: Generated new TSK 0
I20260924 03:12:14.396593 19036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:14.400327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--builtin_ntp_servers=127.0.98.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:14.475693 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:14.475900 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:14.475934 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:14.476979 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:14.477032 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:14.477067 19048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:14.478319 19048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:14.478542 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:14.478794 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:14.483461 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.483462 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.483649 19048 server_base.cc:1034] running on GCE node
W20260924 03:12:14.483777 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.483986 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:14.484220 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:14.485383 19048 hybrid_clock.cc:648] HybridClock initialized: now 1790219534485344 us; error 31 us; skew 500 ppm
I20260924 03:12:14.486603 19048 webserver.cc:460] Webserver started at http://127.0.98.1:44689/ using document root <none> and password file <none>
I20260924 03:12:14.486812 19048 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:14.486884 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:14.487016 19048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:14.487929 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "89e44032e1a341baacd2acd24b2ef3e6"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "3360b3c823b6a01b9574ecf7ce899739"
server_key_iv: "aa9dc3c6bfbcff62b2af566f244cb225"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.488282 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "89e44032e1a341baacd2acd24b2ef3e6"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "3360b3c823b6a01b9574ecf7ce899739"
server_key_iv: "aa9dc3c6bfbcff62b2af566f244cb225"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.489501 19048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:14.490351 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.490620 19048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:14.490720 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "89e44032e1a341baacd2acd24b2ef3e6"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "3360b3c823b6a01b9574ecf7ce899739"
server_key_iv: "aa9dc3c6bfbcff62b2af566f244cb225"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.490796 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.507032 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:14.507261 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.507390 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.507601 19048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:14.507900 19048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:14.507932 19048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.507987 19048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.508013 19048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.513270 19048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46261
I20260924 03:12:14.513343 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46261 every 8 connection(s)
I20260924 03:12:14.513623 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:14.514884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19048
I20260924 03:12:14.515007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:14.515203 392 external_mini_cluster.cc:1428] Setting key 194a99e2099c8a31bf5ec6dde4a3bd13
I20260924 03:12:14.516059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--builtin_ntp_servers=127.0.98.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:12:14.519176 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.514298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37229 (local address 127.0.98.62:44405)
0924 03:12:14.514336 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.514338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.514535 (+ 197us) server_negotiation.cc:407] Connection header received
0924 03:12:14.514944 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.514946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.514962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.515030 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.515918 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.516474 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.517709 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.518011 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.518083 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.518835 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.518836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.518837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:14.518841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.518856 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:14.518860 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.518915 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.519031 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.519059 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:14.519241 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.514290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.98.1:37229)
0924 03:12:14.514432 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.514461 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.514858 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.515158 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.515163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.515172 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.515828 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.515835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.516556 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.516558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.517629 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.517632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.518089 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.518090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.518225 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.518710 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:14.518710 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.518736 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:14.518757 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.518937 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.518942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.518945 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.519015 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.519088 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 03:12:14.519727 19176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44405
I20260924 03:12:14.519822 19176 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.519986 19176 heartbeater.cc:511] Master 127.0.98.62:44405 requested a full tablet report, sending...
I20260924 03:12:14.520445 18988 ts_manager.cc:194] Registered new tserver with Master: 89e44032e1a341baacd2acd24b2ef3e6 (127.0.98.1:46261)
I20260924 03:12:14.521032 18988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37229
W20260924 03:12:14.596508 19179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:14.596683 19179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:14.596701 19179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:14.597795 19179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:14.597834 19179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:14.597883 19179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:14.599222 19179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:14.599421 19179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:14.599674 19179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:14.604333 19187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.604365 19185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.604449 19179 server_base.cc:1034] running on GCE node
W20260924 03:12:14.604393 19184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.604802 19179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:14.605026 19179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:14.606159 19179 hybrid_clock.cc:648] HybridClock initialized: now 1790219534606148 us; error 27 us; skew 500 ppm
I20260924 03:12:14.607389 19179 webserver.cc:460] Webserver started at http://127.0.98.2:36269/ using document root <none> and password file <none>
I20260924 03:12:14.607594 19179 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:14.607635 19179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:14.607761 19179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:14.608666 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "d0bfe2e22b2748a2b049163d50fbe645"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "c00ab8fe33c294f408800befcbc83cbb"
server_key_iv: "892ae2240a23d2b8a9c94637407eebf8"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.608985 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d0bfe2e22b2748a2b049163d50fbe645"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "c00ab8fe33c294f408800befcbc83cbb"
server_key_iv: "892ae2240a23d2b8a9c94637407eebf8"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.610245 19179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:14.610971 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.611168 19179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:14.611234 19179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "d0bfe2e22b2748a2b049163d50fbe645"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "c00ab8fe33c294f408800befcbc83cbb"
server_key_iv: "892ae2240a23d2b8a9c94637407eebf8"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.611312 19179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.634428 19179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:14.634657 19179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.634778 19179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.634974 19179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:14.635246 19179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:14.635275 19179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.635320 19179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.635345 19179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.640501 19179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33055
I20260924 03:12:14.640554 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33055 every 8 connection(s)
I20260924 03:12:14.640847 19179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:14.645293 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.641565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44059 (local address 127.0.98.62:44405)
0924 03:12:14.641609 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.641611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.641815 (+ 204us) server_negotiation.cc:407] Connection header received
0924 03:12:14.642287 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.642288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.642297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.642349 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.642895 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.643349 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.644180 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.644364 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.644424 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.645086 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.645087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.645088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:14.645092 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.645104 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:14.645107 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.645149 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.645216 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.645238 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:12:14.645444 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.641519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.98.2:44059)
0924 03:12:14.641698 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.641734 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.642167 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.642395 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.642397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.642405 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.642832 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.642835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.643414 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.643415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.644116 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.644118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.644433 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.644433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.644545 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.644968 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:14.644969 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.644995 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:14.645014 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.645157 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.645159 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.645161 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.645206 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.645270 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260924 03:12:14.645943 19307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44405
I20260924 03:12:14.646013 19307 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.646135 19307 heartbeater.cc:511] Master 127.0.98.62:44405 requested a full tablet report, sending...
I20260924 03:12:14.646487 18988 ts_manager.cc:194] Registered new tserver with Master: d0bfe2e22b2748a2b049163d50fbe645 (127.0.98.2:33055)
I20260924 03:12:14.646800 18988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44059
I20260924 03:12:14.650969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19179
I20260924 03:12:14.651077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:14.651238 392 external_mini_cluster.cc:1428] Setting key ea2092d419e8bede22aa21c5e1e21691
I20260924 03:12:14.652086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--builtin_ntp_servers=127.0.98.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:14.726833 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:14.726999 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:14.727020 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:14.728009 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:14.728040 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:14.728063 19310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:14.729251 19310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:14.729401 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:14.729595 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:14.733968 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.734094 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:14.734021 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:14.734396 19310 server_base.cc:1034] running on GCE node
I20260924 03:12:14.734647 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:14.734858 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:14.736006 19310 hybrid_clock.cc:648] HybridClock initialized: now 1790219534735992 us; error 25 us; skew 500 ppm
I20260924 03:12:14.737092 19310 webserver.cc:460] Webserver started at http://127.0.98.3:44693/ using document root <none> and password file <none>
I20260924 03:12:14.737326 19310 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:14.737390 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:14.737516 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:14.738555 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b10c91379e943649db54976779a800d"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "ab183628193f9dc944f8d2c7fec41d91"
server_key_iv: "57f0e997811a2ee87b627cd1bdc475a2"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.738893 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b10c91379e943649db54976779a800d"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "ab183628193f9dc944f8d2c7fec41d91"
server_key_iv: "57f0e997811a2ee87b627cd1bdc475a2"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.740082 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:14.740787 19324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.740955 19310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:14.741046 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "8b10c91379e943649db54976779a800d"
format_stamp: "Formatted at 2026-09-24 03:12:14 on dist-test-slave-jtnz"
server_key: "ab183628193f9dc944f8d2c7fec41d91"
server_key_iv: "57f0e997811a2ee87b627cd1bdc475a2"
server_key_version: "encryptionkey@0"
I20260924 03:12:14.741128 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:14.757743 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:14.757980 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:14.758112 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:14.758348 19310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:14.758667 19310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:14.758719 19310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.758769 19310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:14.758798 19310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:14.764209 19310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37765
I20260924 03:12:14.764256 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37765 every 8 connection(s)
I20260924 03:12:14.764541 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:14.766238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19310
I20260924 03:12:14.766350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:14.766535 392 external_mini_cluster.cc:1428] Setting key 81321c023315b7e36ed2f8edd4ee37bb
I20260924 03:12:14.769501 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.765236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38413 (local address 127.0.98.62:44405)
0924 03:12:14.765277 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.765279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.765477 (+ 198us) server_negotiation.cc:407] Connection header received
0924 03:12:14.765998 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.765999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.766008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.766054 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.766862 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.767561 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.768423 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.768600 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.768646 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.769294 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.769295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.769296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:14.769299 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.769313 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:14.769316 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.769369 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.769425 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.769449 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:14.769681 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.765195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.98.3:38413)
0924 03:12:14.765376 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.765411 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.765917 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.766122 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.766125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.766132 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:14.766778 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.766785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.767635 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.767636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.768353 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.768355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.768665 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.768666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.768761 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.769177 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:14.769177 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:14.769202 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:14.769220 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:14.769377 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:14.769380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:14.769382 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.769427 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.769494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 03:12:14.770138 19438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44405
I20260924 03:12:14.770255 19438 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:14.770412 19438 heartbeater.cc:511] Master 127.0.98.62:44405 requested a full tablet report, sending...
I20260924 03:12:14.770771 18988 ts_manager.cc:194] Registered new tserver with Master: 8b10c91379e943649db54976779a800d (127.0.98.3:37765)
I20260924 03:12:14.771124 18988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38413
I20260924 03:12:14.778043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:14.781666 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.779267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.0.98.62:44405)
0924 03:12:14.779324 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.779325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.779406 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:12:14.779458 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.779459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.779468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.779501 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:14.779611 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.780009 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.780653 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.780816 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.780862 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.781492 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:14.781493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:14.781493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:14.781496 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:14.781506 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:14.781508 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:14.781564 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.781609 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.781617 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:12:14.783196 18988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:14.784629 18988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:14.790141 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46261 (local address 127.0.0.1:49188)
0924 03:12:14.786958 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.786973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.787011 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.787347 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.787348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.787350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.787382 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.787384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788482 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.788483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.789176 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.789178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.789862 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.789863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.789949 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.789962 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.790025 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.790072 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260924 03:12:14.790151 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.0.98.1:46261)
0924 03:12:14.787143 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.787146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.787152 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:12:14.787178 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.787179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.787208 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.787281 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.787459 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788402 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.789267 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.789768 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.789857 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.789956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.790036 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.790079 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":92,"threads_started":1}
I20260924 03:12:14.790541 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37765 (local address 127.0.0.1:48176)
0924 03:12:14.786655 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.786691 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.786750 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.787097 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.787099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.787102 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.787156 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.787158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788352 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.788352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.789347 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.789350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.790293 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.790295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.790358 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.790361 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.790419 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.790459 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":62,"spinlock_wait_cycles":768,"thread_start_us":86,"threads_started":1}
I20260924 03:12:14.790565 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.0.98.3:37765)
0924 03:12:14.786964 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.786967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.786974 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:14.786996 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.786997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.787007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.787063 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.787237 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788274 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.789440 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.790100 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.790154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.790253 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.790446 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.790485 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260924 03:12:14.790918 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33055 (local address 127.0.0.1:45448)
0924 03:12:14.786784 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.786793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.786830 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.787409 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.787410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.787412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.787452 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.787454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788213 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.788214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.789349 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.789350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.790339 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.790340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.790390 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.790392 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.790432 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.790461 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":34,"thread_start_us":44,"threads_started":1}
I20260924 03:12:14.791031 19110 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad99317bde34520980c092dd5af6990 (DEFAULT_TABLE table=test-table [id=c4291ca0a3e74d9dad9ef65ac9386f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.791180 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.786617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.0.98.2:33055)
0924 03:12:14.787132 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.787136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.787147 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:12:14.787180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.787182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.787195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.787322 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.787550 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.788147 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.789441 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.790088 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.790154 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.790546 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.790571 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.791124 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":89,"threads_started":1}
I20260924 03:12:14.791213 19372 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad99317bde34520980c092dd5af6990 (DEFAULT_TABLE table=test-table [id=c4291ca0a3e74d9dad9ef65ac9386f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.791397 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad99317bde34520980c092dd5af6990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.791495 19372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad99317bde34520980c092dd5af6990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.794451 19453 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: Bootstrap starting.
I20260924 03:12:14.795068 19453 tablet_bootstrap.cc:654] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.795010 19241 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad99317bde34520980c092dd5af6990 (DEFAULT_TABLE table=test-table [id=c4291ca0a3e74d9dad9ef65ac9386f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:14.795329 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad99317bde34520980c092dd5af6990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:14.795382 19453 log.cc:824] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.795684 19452 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: Bootstrap starting.
I20260924 03:12:14.796000 19453 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: No bootstrap required, opened a new log
I20260924 03:12:14.796082 19453 ts_tablet_manager.cc:1397] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.796257 19452 tablet_bootstrap.cc:654] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.796541 19452 log.cc:824] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.797204 19452 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: No bootstrap required, opened a new log
I20260924 03:12:14.797261 19452 ts_tablet_manager.cc:1397] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.798002 19453 raft_consensus.cc:348] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.798139 19453 raft_consensus.cc:374] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.798178 19453 raft_consensus.cc:729] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e44032e1a341baacd2acd24b2ef3e6, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.798321 19453 consensus_queue.cc:260] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.798579 19453 ts_tablet_manager.cc:1428] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:12:14.798625 19176 heartbeater.cc:503] Master 127.0.98.62:44405 was elected leader, sending a full tablet report...
I20260924 03:12:14.799223 19452 raft_consensus.cc:348] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.799350 19452 raft_consensus.cc:374] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.799392 19452 raft_consensus.cc:729] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b10c91379e943649db54976779a800d, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.799492 19452 consensus_queue.cc:260] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.799733 19452 ts_tablet_manager.cc:1428] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:12:14.799762 19438 heartbeater.cc:503] Master 127.0.98.62:44405 was elected leader, sending a full tablet report...
I20260924 03:12:14.799903 19456 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Bootstrap starting.
I20260924 03:12:14.800426 19456 tablet_bootstrap.cc:654] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:14.800701 19456 log.cc:824] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:14.801326 19456 tablet_bootstrap.cc:492] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: No bootstrap required, opened a new log
I20260924 03:12:14.801398 19456 ts_tablet_manager.cc:1397] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:14.803130 19456 raft_consensus.cc:348] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.803269 19456 raft_consensus.cc:374] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:14.803308 19456 raft_consensus.cc:729] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bfe2e22b2748a2b049163d50fbe645, State: Initialized, Role: FOLLOWER
I20260924 03:12:14.803402 19456 consensus_queue.cc:260] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.803591 19456 ts_tablet_manager.cc:1428] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:14.803642 19307 heartbeater.cc:503] Master 127.0.98.62:44405 was elected leader, sending a full tablet report...
I20260924 03:12:14.824707 19458 raft_consensus.cc:482] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:14.824766 19458 raft_consensus.cc:504] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.825052 19458 leader_election.cc:288] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0bfe2e22b2748a2b049163d50fbe645 (127.0.98.2:33055), 89e44032e1a341baacd2acd24b2ef3e6 (127.0.98.1:46261)
I20260924 03:12:14.827215 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.825211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44529 (local address 127.0.98.2:33055)
0924 03:12:14.825242 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.825244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.825251 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:14.825273 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.825273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.825280 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.825323 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.825453 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.825823 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.826641 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.826935 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.827010 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.827017 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.827138 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.827167 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:14.827216 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.825143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33055 (local address 127.0.98.3:44529)
0924 03:12:14.825186 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.825196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.825231 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.825327 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.825328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.825329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.825392 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.825393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.825889 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.825890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.826567 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.826569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.827013 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.827014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.827074 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.827076 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.827132 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.827163 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:12:14.827415 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.825201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40517 (local address 127.0.98.1:46261)
0924 03:12:14.825235 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.825236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.825347 (+ 111us) server_negotiation.cc:407] Connection header received
0924 03:12:14.825432 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.825433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.825441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.825478 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:14.825608 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.825967 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.826808 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.827104 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.827178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.827184 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.827324 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.827364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:14.827457 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad99317bde34520980c092dd5af6990" candidate_uuid: "8b10c91379e943649db54976779a800d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bfe2e22b2748a2b049163d50fbe645" is_pre_election: true
I20260924 03:12:14.827585 19261 raft_consensus.cc:2454] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b10c91379e943649db54976779a800d in term 0.
I20260924 03:12:14.827605 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.825162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46261 (local address 127.0.98.3:40517)
0924 03:12:14.825284 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.825292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:14.825373 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:14.825488 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:14.825489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:14.825490 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:14.825544 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.825547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.826036 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.826037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.826729 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:14.826731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.827180 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.827181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:14.827245 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.827247 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:14.827504 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 03:12:14.827543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260924 03:12:14.827780 19327 leader_election.cc:302] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b10c91379e943649db54976779a800d, d0bfe2e22b2748a2b049163d50fbe645; no voters:
I20260924 03:12:14.827886 19458 raft_consensus.cc:2781] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:14.827865 19130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad99317bde34520980c092dd5af6990" candidate_uuid: "8b10c91379e943649db54976779a800d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e44032e1a341baacd2acd24b2ef3e6" is_pre_election: true
I20260924 03:12:14.827925 19458 raft_consensus.cc:482] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:14.827943 19458 raft_consensus.cc:3037] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.827961 19130 raft_consensus.cc:2454] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b10c91379e943649db54976779a800d in term 0.
I20260924 03:12:14.828505 19458 raft_consensus.cc:504] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.828629 19458 leader_election.cc:288] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [CANDIDATE]: Term 1 election: Requested vote from peers d0bfe2e22b2748a2b049163d50fbe645 (127.0.98.2:33055), 89e44032e1a341baacd2acd24b2ef3e6 (127.0.98.1:46261)
I20260924 03:12:14.828748 19130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad99317bde34520980c092dd5af6990" candidate_uuid: "8b10c91379e943649db54976779a800d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e44032e1a341baacd2acd24b2ef3e6"
I20260924 03:12:14.828763 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad99317bde34520980c092dd5af6990" candidate_uuid: "8b10c91379e943649db54976779a800d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bfe2e22b2748a2b049163d50fbe645"
I20260924 03:12:14.828817 19261 raft_consensus.cc:3037] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.828833 19130 raft_consensus.cc:3037] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:14.829360 19261 raft_consensus.cc:2454] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b10c91379e943649db54976779a800d in term 1.
I20260924 03:12:14.829487 19130 raft_consensus.cc:2454] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b10c91379e943649db54976779a800d in term 1.
I20260924 03:12:14.829522 19327 leader_election.cc:302] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b10c91379e943649db54976779a800d, d0bfe2e22b2748a2b049163d50fbe645; no voters:
I20260924 03:12:14.829639 19458 raft_consensus.cc:2781] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:14.829773 19458 raft_consensus.cc:686] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 1 LEADER]: Becoming Leader. State: Replica: 8b10c91379e943649db54976779a800d, State: Running, Role: LEADER
I20260924 03:12:14.829825 19458 consensus_queue.cc:237] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } }
I20260924 03:12:14.830389 18988 catalog_manager.cc:5495] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b10c91379e943649db54976779a800d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8b10c91379e943649db54976779a800d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b10c91379e943649db54976779a800d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:14.837460 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.835643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.98.3:37765)
0924 03:12:14.835693 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:14.835694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:14.835703 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:14.835734 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:14.835735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:14.835748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:14.835804 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:14.835898 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.836356 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.836964 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:14.837116 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:14.837162 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:14.837243 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:14.837324 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:14.837380 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:14.837399 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:12:14.837404 (+ 5us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:12:14.838892 19261 raft_consensus.cc:1261] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 1 FOLLOWER]: Refusing update from remote peer 8b10c91379e943649db54976779a800d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:14.838922 19130 raft_consensus.cc:1261] T 2ad99317bde34520980c092dd5af6990 P 89e44032e1a341baacd2acd24b2ef3e6 [term 1 FOLLOWER]: Refusing update from remote peer 8b10c91379e943649db54976779a800d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:14.839200 19462 consensus_queue.cc:1035] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0bfe2e22b2748a2b049163d50fbe645" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:14.839288 19458 consensus_queue.cc:1035] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e44032e1a341baacd2acd24b2ef3e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:14.841207 19465 mvcc.cc:205] Tried to move back new op lower bound from 7332739214698094592 to 7332739214663462912. Current Snapshot: MvccSnapshot[applied={T|T < 7332739214698094592}]
I20260924 03:12:14.841367 19466 mvcc.cc:205] Tried to move back new op lower bound from 7332739214698094592 to 7332739214663462912. Current Snapshot: MvccSnapshot[applied={T|T < 7332739214698094592}]
I20260924 03:12:14.843529 18988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:14.843607 18988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:14.844177 18988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5a53d9bfc1748b28e288719526567df: Sending DeleteTablet for 3 replicas of tablet 2ad99317bde34520980c092dd5af6990
I20260924 03:12:14.844513 19241 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad99317bde34520980c092dd5af6990 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='slave'} at 127.0.0.1:45448
I20260924 03:12:14.844498 19372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad99317bde34520980c092dd5af6990 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='slave'} at 127.0.0.1:48176
I20260924 03:12:14.844648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19048
I20260924 03:12:14.844702 19110 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad99317bde34520980c092dd5af6990 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:14 UTC) from {username='slave'} at 127.0.0.1:49188
I20260924 03:12:14.845513 19475 tablet_replica.cc:330] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: stopping tablet replica
I20260924 03:12:14.845592 19475 raft_consensus.cc:2229] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:14.845671 19475 raft_consensus.cc:2258] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:14.846060 19475 ts_tablet_manager.cc:1905] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:14.847398 19475 ts_tablet_manager.cc:1918] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:14.847467 19475 log.cc:1196] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/2ad99317bde34520980c092dd5af6990
I20260924 03:12:14.847697 19475 ts_tablet_manager.cc:1939] T 2ad99317bde34520980c092dd5af6990 P d0bfe2e22b2748a2b049163d50fbe645: Deleting consensus metadata
I20260924 03:12:14.848023 18974 catalog_manager.cc:4841] TS d0bfe2e22b2748a2b049163d50fbe645 (127.0.98.2:33055): tablet 2ad99317bde34520980c092dd5af6990 (table test-table [id=c4291ca0a3e74d9dad9ef65ac9386f33]) successfully deleted
W20260924 03:12:14.849725 18972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46261 (error 108)
I20260924 03:12:14.849988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19179
I20260924 03:12:14.850132 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.850001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46261 (local address 127.0.0.1:49190)
0924 03:12:14.850063 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.850100 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 03:12:14.851686 18972 catalog_manager.cc:4584] TS 89e44032e1a341baacd2acd24b2ef3e6 (127.0.98.1:46261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ad99317bde34520980c092dd5af6990: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46261: connect: Connection refused (error 111)
I20260924 03:12:14.854552 19476 tablet_replica.cc:330] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d: stopping tablet replica
I20260924 03:12:14.854619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19310
I20260924 03:12:14.854661 19476 raft_consensus.cc:2229] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:14.854758 19476 raft_consensus.cc:2258] T 2ad99317bde34520980c092dd5af6990 P 8b10c91379e943649db54976779a800d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:14.858847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 18957
I20260924 03:12:14.858880 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:14.858783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37765 (local address 127.0.0.1:48190)
0924 03:12:14.858831 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:14.858861 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:37765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
2026-09-24T03:12:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (656 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:14 dist-test-slave-jtnz krb5kdc[19480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:14 dist-test-slave-jtnz krb5kdc[19480](info): set up 2 sockets
Sep 24 03:12:14 dist-test-slave-jtnz krb5kdc[19480](info): commencing operation
krb5kdc: starting...
W20260924 03:12:15.813802 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:15 dist-test-slave-jtnz krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:15.917848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:15.989189 19493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:15.989420 19493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:15.989452 19493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:15.990594 19493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:15.990650 19493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:15.990677 19493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:15.990697 19493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:15.990746 19493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:15.992175 19493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:15.992415 19493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:15.992617 19493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:15.996820 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:15.996845 19493 server_base.cc:1034] running on GCE node
W20260924 03:12:15.996820 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:15.996802 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:15.997238 19493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:15.997485 19493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:15.998611 19493 hybrid_clock.cc:648] HybridClock initialized: now 1790219535998599 us; error 26 us; skew 500 ppm
Sep 24 03:12:15 dist-test-slave-jtnz krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:16.000824 19493 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:16.001180 19493 webserver.cc:460] Webserver started at http://127.0.98.62:33907/ using document root <none> and password file <none>
I20260924 03:12:16.001401 19493 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:16.001466 19493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:16.001610 19493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:16.002624 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "4fa3960c440f4d948e88726e17112b59"
format_stamp: "Formatted at 2026-09-24 03:12:15 on dist-test-slave-jtnz"
server_key: "b3ab97e6561404ce9a98b9bdd2db7d82"
server_key_iv: "7dd156c6d4ff9264f36138dc049737d7"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.002954 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "4fa3960c440f4d948e88726e17112b59"
format_stamp: "Formatted at 2026-09-24 03:12:15 on dist-test-slave-jtnz"
server_key: "b3ab97e6561404ce9a98b9bdd2db7d82"
server_key_iv: "7dd156c6d4ff9264f36138dc049737d7"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.004171 19493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:16.004830 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.004997 19493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:16.005096 19493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "4fa3960c440f4d948e88726e17112b59"
format_stamp: "Formatted at 2026-09-24 03:12:15 on dist-test-slave-jtnz"
server_key: "b3ab97e6561404ce9a98b9bdd2db7d82"
server_key_iv: "7dd156c6d4ff9264f36138dc049737d7"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.005172 19493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:16.019381 19493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:16.020879 19493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:16.021039 19493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:16.024464 19493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42179
I20260924 03:12:16.024519 19560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42179 every 8 connection(s)
I20260924 03:12:16.024811 19493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:16.025497 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:16.027444 19561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: Bootstrap starting.
I20260924 03:12:16.027982 19561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:16.028258 19561 log.cc:824] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:16.028877 19561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: No bootstrap required, opened a new log
I20260924 03:12:16.030336 19561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } }
I20260924 03:12:16.030452 19561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:16.030471 19561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa3960c440f4d948e88726e17112b59, State: Initialized, Role: FOLLOWER
I20260924 03:12:16.030663 19561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } }
I20260924 03:12:16.030740 19561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:16.030784 19561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:16.030830 19561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:16.031286 19561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } }
I20260924 03:12:16.031392 19561 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa3960c440f4d948e88726e17112b59; no voters:
I20260924 03:12:16.031581 19561 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:16.031630 19564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:16.031702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19493
I20260924 03:12:16.031771 19564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa3960c440f4d948e88726e17112b59, State: Running, Role: LEADER
I20260924 03:12:16.031802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:16.031807 19561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:16.031850 19564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } }
I20260924 03:12:16.032009 392 external_mini_cluster.cc:1428] Setting key 9981bdcc7c3e2ee4b0b29397f8f157a8
I20260924 03:12:16.032148 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa3960c440f4d948e88726e17112b59. Latest consensus state: current_term: 1 leader_uuid: "4fa3960c440f4d948e88726e17112b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } } }
I20260924 03:12:16.032241 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:16.032394 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa3960c440f4d948e88726e17112b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa3960c440f4d948e88726e17112b59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42179 } } }
I20260924 03:12:16.032480 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:16.032805 19572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:16.033362 19572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:16.034993 19572 catalog_manager.cc:1347] Generated new cluster ID: 8881715952844bd7b0a617a2377cbbdd
I20260924 03:12:16.035040 19572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:16.043071 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.032590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.0.98.62:42179)
0924 03:12:16.032778 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.032788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.032824 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:12:16.032947 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.032955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.033013 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.033710 (+ 697us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.033911 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.035260 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.036224 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.036762 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.036840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.039496 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.039500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.039501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.039511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.041463 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.041842 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.041845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.041845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.041955 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.042214 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.042214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.042215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.042494 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.042651 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.042741 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.042866 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260924 03:12:16.048877 19572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:16.049551 19572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:16.053009 19572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: Generated new TSK 0
I20260924 03:12:16.053164 19572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:12:16.082135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:16.155464 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:16.155629 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:16.155649 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:16.156672 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:16.156709 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:16.156756 19587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:16.158209 19587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:16.158411 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:16.158594 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:16.162911 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.162921 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.162915 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:16.163198 19587 server_base.cc:1034] running on GCE node
I20260924 03:12:16.163407 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:16.163621 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:16.164776 19587 hybrid_clock.cc:648] HybridClock initialized: now 1790219536164764 us; error 23 us; skew 500 ppm
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:16.167273 19587 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:12:16.167670 19587 webserver.cc:460] Webserver started at http://127.0.98.1:36921/ using document root <none> and password file <none>
I20260924 03:12:16.167842 19587 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:16.167878 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:16.167958 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:16.168814 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "c432a9632d3c43a093633d65a4779e1b"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "6a5097464d05957c14274a28567ac450"
server_key_iv: "5e45091db290feb0c8d2bb608d1518a6"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.169088 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c432a9632d3c43a093633d65a4779e1b"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "6a5097464d05957c14274a28567ac450"
server_key_iv: "5e45091db290feb0c8d2bb608d1518a6"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.170265 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:16.170933 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.171090 19587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:16.171170 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "c432a9632d3c43a093633d65a4779e1b"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "6a5097464d05957c14274a28567ac450"
server_key_iv: "5e45091db290feb0c8d2bb608d1518a6"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.171242 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:16.183279 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:16.184772 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:16.184907 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:16.185089 19587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:16.185340 19587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:16.185369 19587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.185386 19587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:16.185425 19587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.190519 19587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45835
I20260924 03:12:16.190601 19715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45835 every 8 connection(s)
I20260924 03:12:16.190881 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:16.195660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19587
I20260924 03:12:16.195766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:16.195950 392 external_mini_cluster.cc:1428] Setting key 407abd6c672fbf563e0d60027c50ee7a
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:16.199864 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.191484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42179 (local address 127.0.98.1:58097)
0924 03:12:16.191627 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.191664 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.192469 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.192653 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.192656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.192867 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:16.193034 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.193038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.193648 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.193649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.194451 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.194453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.194834 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.194835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.194937 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.195298 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:16.195305 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:16.196735 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:16.198501 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.198508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.198510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.198818 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.199013 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.199014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.199014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.199155 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.199438 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:16.199442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:16.199550 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.199627 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.199731 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260924 03:12:16.199898 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.191529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58097 (local address 127.0.98.62:42179)
0924 03:12:16.191571 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.191573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.191733 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:12:16.192552 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.192552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.192560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.192608 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.193107 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.193553 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.194540 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.194746 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.194815 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.196817 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.196834 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.196835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.196843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.198428 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.198901 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.198902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.198902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.198950 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.199224 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.199225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.199225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.199374 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.199438 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.199673 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.199762 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:16.200344 19716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42179
I20260924 03:12:16.200436 19716 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:16.200608 19716 heartbeater.cc:511] Master 127.0.98.62:42179 requested a full tablet report, sending...
I20260924 03:12:16.201136 19525 ts_manager.cc:194] Registered new tserver with Master: c432a9632d3c43a093633d65a4779e1b (127.0.98.1:45835)
I20260924 03:12:16.201741 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58097
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:12:16.221067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:16.296132 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:16.296303 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:16.296339 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:16.297406 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:16.297437 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:16.297461 19721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:16.298769 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:16.298930 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:16.299077 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:16.303570 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.303524 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.303527 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:16.303833 19721 server_base.cc:1034] running on GCE node
I20260924 03:12:16.304013 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:16.304177 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:16.305325 19721 hybrid_clock.cc:648] HybridClock initialized: now 1790219536305306 us; error 30 us; skew 500 ppm
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:16.307621 19721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:12:16.308041 19721 webserver.cc:460] Webserver started at http://127.0.98.2:39305/ using document root <none> and password file <none>
I20260924 03:12:16.308202 19721 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:16.308238 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:16.308360 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:16.309281 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "f0818a09e347c511bd0bbe2760c67f42"
server_key_iv: "30445cd43e8c73b1fcc6831435ae1ea9"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.309552 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "f0818a09e347c511bd0bbe2760c67f42"
server_key_iv: "30445cd43e8c73b1fcc6831435ae1ea9"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.310730 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:16.311347 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.311524 19721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:16.311627 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "f0818a09e347c511bd0bbe2760c67f42"
server_key_iv: "30445cd43e8c73b1fcc6831435ae1ea9"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.311690 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:16.321087 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:16.322745 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:16.322877 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:16.323055 19721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:16.323304 19721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:16.323333 19721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.323351 19721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:16.323393 19721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.328476 19721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39799
I20260924 03:12:16.328572 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39799 every 8 connection(s)
I20260924 03:12:16.328827 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:16.334892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19721
I20260924 03:12:16.334995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:16.335191 392 external_mini_cluster.cc:1428] Setting key daaba023c96def3b9721940d4aec5568
I20260924 03:12:16.337908 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.329418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42179 (local address 127.0.98.2:38169)
0924 03:12:16.329559 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.329593 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.330471 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.330677 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.330680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.330896 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:16.331061 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.331064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.331675 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.331676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.332550 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.332552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.332939 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.332940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.333066 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.333522 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:16.333530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:16.334814 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:16.336489 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.336495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.336496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.336829 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.337016 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.337017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.337017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.337179 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.337473 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:16.337477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:16.337583 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.337662 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.337764 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260924 03:12:16.337922 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.329452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38169 (local address 127.0.98.62:42179)
0924 03:12:16.329493 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.329495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.329675 (+ 180us) server_negotiation.cc:407] Connection header received
0924 03:12:16.330554 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.330555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.330577 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.330624 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.331125 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.331588 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.332621 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.332860 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.332907 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.334908 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.334909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.334910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.334917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.336391 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.336903 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.336904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.336904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.336954 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.337258 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.337259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.337259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.337402 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.337474 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.337708 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.337798 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:16.338454 19850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42179
I20260924 03:12:16.338537 19850 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:16.338708 19850 heartbeater.cc:511] Master 127.0.98.62:42179 requested a full tablet report, sending...
I20260924 03:12:16.339035 19525 ts_manager.cc:194] Registered new tserver with Master: c6d9f6f4f2734c3a93bc4b7ea505fcf8 (127.0.98.2:39799)
I20260924 03:12:16.339522 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38169
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:12:16.358246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:16.429044 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:16.429226 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:16.429261 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:16.430362 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:16.430423 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:16.430466 19855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:16.431994 19855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:16.432264 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:16.432487 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:16.436869 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.436903 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.436913 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:16.436964 19855 server_base.cc:1034] running on GCE node
I20260924 03:12:16.437213 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:16.437378 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:16.438503 19855 hybrid_clock.cc:648] HybridClock initialized: now 1790219536438494 us; error 31 us; skew 500 ppm
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:16.440738 19855 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:12:16.441090 19855 webserver.cc:460] Webserver started at http://127.0.98.3:37365/ using document root <none> and password file <none>
I20260924 03:12:16.441242 19855 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:16.441298 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:16.441434 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:16.442391 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "a7a207b272b546e5809fef1432160657"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "35d8daa6221636d4f883fbe7f6a6e95b"
server_key_iv: "14060adf44b41a2bfffa7e6c84c02730"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.442662 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a7a207b272b546e5809fef1432160657"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "35d8daa6221636d4f883fbe7f6a6e95b"
server_key_iv: "14060adf44b41a2bfffa7e6c84c02730"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.443733 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:16.444298 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.444468 19855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:16.444545 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "a7a207b272b546e5809fef1432160657"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "35d8daa6221636d4f883fbe7f6a6e95b"
server_key_iv: "14060adf44b41a2bfffa7e6c84c02730"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.444607 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:16.457731 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:16.459250 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:16.459388 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:16.459554 19855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:16.459789 19855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:16.459815 19855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.459833 19855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:16.459870 19855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.465281 19855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39407
I20260924 03:12:16.465353 19983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39407 every 8 connection(s)
I20260924 03:12:16.465627 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:12:16 dist-test-slave-jtnz krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:16.471827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 19855
I20260924 03:12:16.471944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:16.472132 392 external_mini_cluster.cc:1428] Setting key 1ff2f08c083c1cfed2a9d1cddc8cc371
I20260924 03:12:16.474761 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.466244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47849 (local address 127.0.98.62:42179)
0924 03:12:16.466287 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.466289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.466517 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:12:16.467422 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.467424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.467437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.467483 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.468065 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.468547 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.469512 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.469737 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.469821 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.471861 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.471863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.471863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.471871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.473326 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.473787 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.473788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.473788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.473838 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.474096 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.474097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.474097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.474257 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.474334 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.474592 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.474670 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:12:16.474943 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.466245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42179 (local address 127.0.98.3:47849)
0924 03:12:16.466382 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.466412 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.467312 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.467554 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.467558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.467798 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:16.467970 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.467975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.468657 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.468658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.469422 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.469424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.469829 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.469831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.469952 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.470495 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:16.470503 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:16.471767 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:16.473423 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.473430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.473432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.473670 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.473903 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:16.473904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:16.473905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:16.474016 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:16.474323 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:16.474330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:16.474462 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.474576 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.474723 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260924 03:12:16.475405 19984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42179
I20260924 03:12:16.475479 19984 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:16.475611 19984 heartbeater.cc:511] Master 127.0.98.62:42179 requested a full tablet report, sending...
I20260924 03:12:16.475907 19525 ts_manager.cc:194] Registered new tserver with Master: a7a207b272b546e5809fef1432160657 (127.0.98.3:39407)
I20260924 03:12:16.476276 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47849
I20260924 03:12:16.483593 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:16.489916 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.484934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.0.98.62:42179)
0924 03:12:16.484988 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.484990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.485082 (+ 92us) server_negotiation.cc:407] Connection header received
0924 03:12:16.485131 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.485131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.485138 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.485185 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:16.485667 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.486111 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.486833 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.486997 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.487045 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.488036 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:16.488037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:16.488038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:16.488043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:16.489137 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.489409 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.489410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.489410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.489453 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:16.489563 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:16.489563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:16.489564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:16.489669 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:16.489733 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.489813 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.489864 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:12:16.491487 19525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:16.492910 19525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:16.498500 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.494790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45835 (local address 127.0.0.1:56892)
0924 03:12:16.495148 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.495154 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.495180 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.495400 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.495402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.495404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:16.495462 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.495463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496709 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.496710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.497408 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.497409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498154 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.498272 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498291 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.498375 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.498425 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":80,"threads_started":1}
I20260924 03:12:16.498500 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.495091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.1:45835)
0924 03:12:16.495235 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.495238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.495247 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:16.495283 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.495285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.495300 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.495380 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:16.495556 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496579 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.497489 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498068 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498143 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498234 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.498392 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.498428 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260924 03:12:16.498771 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.494789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39799 (local address 127.0.0.1:42160)
0924 03:12:16.494951 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.494985 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.495053 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.495483 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.495484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.495485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:16.495515 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.495517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496676 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.496676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.497703 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.497705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498542 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.498620 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498625 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.498657 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.498694 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":29,"spinlock_wait_cycles":384,"thread_start_us":83,"threads_started":1}
I20260924 03:12:16.499045 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.494910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.0.98.3:39407)
0924 03:12:16.495052 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.495056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.495264 (+ 208us) server_negotiation.cc:407] Connection header received
0924 03:12:16.495314 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.495315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.495330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.495391 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:16.495598 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496327 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.497689 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498353 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498398 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498446 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.498918 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.498958 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260924 03:12:16.499094 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.494942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.0.98.2:39799)
0924 03:12:16.495335 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.495338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.495344 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:12:16.495400 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.495402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.495415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.495489 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:16.495568 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496572 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.497780 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498461 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498869 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498934 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.498959 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.499013 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":71,"threads_started":1}
I20260924 03:12:16.499264 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.494789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39407 (local address 127.0.0.1:40146)
0924 03:12:16.495222 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.495227 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.495257 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.495467 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.495468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.495471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:16.495515 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.495517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.496425 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.496426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.497599 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.497601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.498508 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.498509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.498821 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.498826 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.499164 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.499205 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":23,"thread_start_us":53,"threads_started":1}
I20260924 03:12:16.499222 19650 tablet_service.cc:1467] Processing CreateTablet for tablet 15ece6505b374ca48072d1a268fab77d (DEFAULT_TABLE table=test-table [id=211510155ff3433b8da2994389a72fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:16.499557 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ece6505b374ca48072d1a268fab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:16.499730 19784 tablet_service.cc:1467] Processing CreateTablet for tablet 15ece6505b374ca48072d1a268fab77d (DEFAULT_TABLE table=test-table [id=211510155ff3433b8da2994389a72fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:16.500023 19784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ece6505b374ca48072d1a268fab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:16.502127 19998 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Bootstrap starting.
I20260924 03:12:16.502770 19918 tablet_service.cc:1467] Processing CreateTablet for tablet 15ece6505b374ca48072d1a268fab77d (DEFAULT_TABLE table=test-table [id=211510155ff3433b8da2994389a72fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:16.502945 19998 tablet_bootstrap.cc:654] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:16.503091 19918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ece6505b374ca48072d1a268fab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:16.503540 19998 log.cc:824] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:16.504242 19998 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: No bootstrap required, opened a new log
I20260924 03:12:16.504310 19998 ts_tablet_manager.cc:1397] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:16.506300 19999 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Bootstrap starting.
I20260924 03:12:16.506409 20001 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Bootstrap starting.
I20260924 03:12:16.506348 19998 raft_consensus.cc:348] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.506495 19998 raft_consensus.cc:374] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:16.506556 19998 raft_consensus.cc:729] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c432a9632d3c43a093633d65a4779e1b, State: Initialized, Role: FOLLOWER
I20260924 03:12:16.506703 19998 consensus_queue.cc:260] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.506912 19999 tablet_bootstrap.cc:654] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:16.506961 19998 ts_tablet_manager.cc:1428] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:16.507053 20001 tablet_bootstrap.cc:654] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:16.507079 19716 heartbeater.cc:503] Master 127.0.98.62:42179 was elected leader, sending a full tablet report...
I20260924 03:12:16.507488 19999 log.cc:824] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:16.507630 20001 log.cc:824] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:16.508363 19999 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: No bootstrap required, opened a new log
I20260924 03:12:16.508412 19999 ts_tablet_manager.cc:1397] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:16.508388 20001 tablet_bootstrap.cc:492] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: No bootstrap required, opened a new log
I20260924 03:12:16.508466 20001 ts_tablet_manager.cc:1397] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:16.509651 19999 raft_consensus.cc:348] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.509788 19999 raft_consensus.cc:374] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:16.509806 19999 raft_consensus.cc:729] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d9f6f4f2734c3a93bc4b7ea505fcf8, State: Initialized, Role: FOLLOWER
I20260924 03:12:16.509757 20001 raft_consensus.cc:348] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.509964 20001 raft_consensus.cc:374] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:16.509933 19999 consensus_queue.cc:260] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.510007 20001 raft_consensus.cc:729] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a207b272b546e5809fef1432160657, State: Initialized, Role: FOLLOWER
I20260924 03:12:16.510115 20001 consensus_queue.cc:260] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.510231 19999 ts_tablet_manager.cc:1428] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:16.510250 19850 heartbeater.cc:503] Master 127.0.98.62:42179 was elected leader, sending a full tablet report...
I20260924 03:12:16.510428 19984 heartbeater.cc:503] Master 127.0.98.62:42179 was elected leader, sending a full tablet report...
I20260924 03:12:16.510617 20001 ts_tablet_manager.cc:1428] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 03:12:16.579322 19851 tablet.cc:2369] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:16.691617 19717 tablet.cc:2369] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:12:16.716087 19985 tablet.cc:2369] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:16.817667 20002 raft_consensus.cc:482] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:16.817788 20002 raft_consensus.cc:504] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.818183 20002 leader_election.cc:288] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7a207b272b546e5809fef1432160657 (127.0.98.3:39407), c6d9f6f4f2734c3a93bc4b7ea505fcf8 (127.0.98.2:39799)
I20260924 03:12:16.820922 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.818456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48571 (local address 127.0.98.3:39407)
0924 03:12:16.818501 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.818503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.818576 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:12:16.818622 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.818623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.818633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.818697 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:16.818843 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.819281 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820233 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.820573 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820623 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.820649 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.820808 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.820852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:12:16.820945 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.818359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39407 (local address 127.0.98.1:48571)
0924 03:12:16.818484 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.818493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.818568 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.818701 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.818702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.818703 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:16.818776 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.818778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.819373 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.819374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.820147 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.820148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.820647 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.820717 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.820725 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.820767 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.820813 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":27,"thread_start_us":55,"threads_started":1}
I20260924 03:12:16.820977 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.818424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39667 (local address 127.0.98.2:39799)
0924 03:12:16.818466 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.818468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.818556 (+ 88us) server_negotiation.cc:407] Connection header received
0924 03:12:16.818625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.818626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.818634 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.818692 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:16.818882 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.819278 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820250 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.820583 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820662 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.820670 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.820861 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.820896 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:16.821046 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.818353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39799 (local address 127.0.98.1:39667)
0924 03:12:16.818475 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.818486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:16.818563 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:16.818712 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:16.818713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:16.818715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:16.818794 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.818795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.819353 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.819354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.820149 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:16.820151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.820681 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.820682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:16.820765 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.820769 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:12:16.820841 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:12:16.820875 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":54,"thread_start_us":64,"threads_started":1}
I20260924 03:12:16.821376 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ece6505b374ca48072d1a268fab77d" candidate_uuid: "c432a9632d3c43a093633d65a4779e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" is_pre_election: true
I20260924 03:12:16.821493 19938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ece6505b374ca48072d1a268fab77d" candidate_uuid: "c432a9632d3c43a093633d65a4779e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a207b272b546e5809fef1432160657" is_pre_election: true
I20260924 03:12:16.821544 19804 raft_consensus.cc:2454] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c432a9632d3c43a093633d65a4779e1b in term 0.
I20260924 03:12:16.821662 19938 raft_consensus.cc:2454] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c432a9632d3c43a093633d65a4779e1b in term 0.
I20260924 03:12:16.821796 19604 leader_election.cc:302] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c432a9632d3c43a093633d65a4779e1b, c6d9f6f4f2734c3a93bc4b7ea505fcf8; no voters:
I20260924 03:12:16.821915 20002 raft_consensus.cc:2781] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:16.821982 20002 raft_consensus.cc:482] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:16.822012 20002 raft_consensus.cc:3037] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:16.822697 20002 raft_consensus.cc:504] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.822847 20002 leader_election.cc:288] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [CANDIDATE]: Term 1 election: Requested vote from peers a7a207b272b546e5809fef1432160657 (127.0.98.3:39407), c6d9f6f4f2734c3a93bc4b7ea505fcf8 (127.0.98.2:39799)
I20260924 03:12:16.822983 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ece6505b374ca48072d1a268fab77d" candidate_uuid: "c432a9632d3c43a093633d65a4779e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8"
I20260924 03:12:16.822992 19938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ece6505b374ca48072d1a268fab77d" candidate_uuid: "c432a9632d3c43a093633d65a4779e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a207b272b546e5809fef1432160657"
I20260924 03:12:16.823088 19804 raft_consensus.cc:3037] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:16.823082 19938 raft_consensus.cc:3037] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:16.823735 19938 raft_consensus.cc:2454] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c432a9632d3c43a093633d65a4779e1b in term 1.
I20260924 03:12:16.823797 19804 raft_consensus.cc:2454] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c432a9632d3c43a093633d65a4779e1b in term 1.
I20260924 03:12:16.823901 19605 leader_election.cc:302] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a207b272b546e5809fef1432160657, c432a9632d3c43a093633d65a4779e1b; no voters:
I20260924 03:12:16.823994 20002 raft_consensus.cc:2781] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:16.824143 20002 raft_consensus.cc:686] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [term 1 LEADER]: Becoming Leader. State: Replica: c432a9632d3c43a093633d65a4779e1b, State: Running, Role: LEADER
I20260924 03:12:16.824234 20002 consensus_queue.cc:237] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } }
I20260924 03:12:16.824970 19524 catalog_manager.cc:5495] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to c432a9632d3c43a093633d65a4779e1b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c432a9632d3c43a093633d65a4779e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c432a9632d3c43a093633d65a4779e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45835 } health_report { overall_health: HEALTHY } } }
I20260924 03:12:16.848026 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.846126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.1:45835)
0924 03:12:16.846213 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:16.846215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:16.846223 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:12:16.846272 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:16.846273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:16.846286 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:16.846361 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:16.846460 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.846847 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.847469 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:16.847616 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:16.847660 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:16.847764 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:16.847837 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:16.847888 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:16.847915 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 03:12:16.847929 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:12:16.849509 19804 raft_consensus.cc:1261] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 1 FOLLOWER]: Refusing update from remote peer c432a9632d3c43a093633d65a4779e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:16.849512 19938 raft_consensus.cc:1261] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 1 FOLLOWER]: Refusing update from remote peer c432a9632d3c43a093633d65a4779e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:16.849824 20009 consensus_queue.cc:1035] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a207b272b546e5809fef1432160657" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:16.849934 20002 consensus_queue.cc:1035] T 15ece6505b374ca48072d1a268fab77d P c432a9632d3c43a093633d65a4779e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d9f6f4f2734c3a93bc4b7ea505fcf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:16.851989 20012 mvcc.cc:205] Tried to move back new op lower bound from 7332739222933381120 to 7332739222832607232. Current Snapshot: MvccSnapshot[applied={T|T < 7332739222933381120}]
I20260924 03:12:16.852191 20014 mvcc.cc:205] Tried to move back new op lower bound from 7332739222933381120 to 7332739222832607232. Current Snapshot: MvccSnapshot[applied={T|T < 7332739222933381120}]
I20260924 03:12:16.854485 19525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:16.854583 19525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:16.855206 19525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fa3960c440f4d948e88726e17112b59: Sending DeleteTablet for 3 replicas of tablet 15ece6505b374ca48072d1a268fab77d
I20260924 03:12:16.855528 19784 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ece6505b374ca48072d1a268fab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42160
I20260924 03:12:16.855511 19918 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ece6505b374ca48072d1a268fab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40146
I20260924 03:12:16.855721 20023 tablet_replica.cc:330] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: stopping tablet replica
I20260924 03:12:16.855721 19650 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ece6505b374ca48072d1a268fab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260924 03:12:16.855842 20023 raft_consensus.cc:2229] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:16.855953 20023 raft_consensus.cc:2258] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:16.856144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19587
I20260924 03:12:16.857156 20023 ts_tablet_manager.cc:1905] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:16.858013 20022 tablet_replica.cc:330] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: stopping tablet replica
I20260924 03:12:16.858114 20022 raft_consensus.cc:2229] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:16.858235 20022 raft_consensus.cc:2258] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:16.858645 20022 ts_tablet_manager.cc:1905] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:16.859134 20023 ts_tablet_manager.cc:1918] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:16.859205 20023 log.cc:1196] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/15ece6505b374ca48072d1a268fab77d
I20260924 03:12:16.859455 20023 ts_tablet_manager.cc:1939] T 15ece6505b374ca48072d1a268fab77d P c6d9f6f4f2734c3a93bc4b7ea505fcf8: Deleting consensus metadata
I20260924 03:12:16.860041 19510 catalog_manager.cc:4841] TS c6d9f6f4f2734c3a93bc4b7ea505fcf8 (127.0.98.2:39799): tablet 15ece6505b374ca48072d1a268fab77d (table test-table [id=211510155ff3433b8da2994389a72fa5]) successfully deleted
I20260924 03:12:16.860231 20022 ts_tablet_manager.cc:1918] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:16.860283 20022 log.cc:1196] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/15ece6505b374ca48072d1a268fab77d
I20260924 03:12:16.860525 20022 ts_tablet_manager.cc:1939] T 15ece6505b374ca48072d1a268fab77d P a7a207b272b546e5809fef1432160657: Deleting consensus metadata
I20260924 03:12:16.861024 19511 catalog_manager.cc:4841] TS a7a207b272b546e5809fef1432160657 (127.0.98.3:39407): tablet 15ece6505b374ca48072d1a268fab77d (table test-table [id=211510155ff3433b8da2994389a72fa5]) successfully deleted
W20260924 03:12:16.861449 19509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45835 (error 108)
I20260924 03:12:16.861639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19721
I20260924 03:12:16.862215 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:16.861762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45835 (local address 127.0.0.1:56918)
0924 03:12:16.862107 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:16.862144 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260924 03:12:16.862830 19509 catalog_manager.cc:4584] TS c432a9632d3c43a093633d65a4779e1b (127.0.98.1:45835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15ece6505b374ca48072d1a268fab77d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45835: connect: Connection refused (error 111)
I20260924 03:12:16.866662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19855
I20260924 03:12:16.871084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 19493
2026-09-24T03:12:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:16Z Disabled control of system clock
I20260924 03:12:16.896899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:16.967199 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:16.967399 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:16.967430 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:16.968403 20029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:16.968459 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:16.968477 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:16.968497 20029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:16.968523 20029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:16.969748 20029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:16.969992 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:16.970279 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:16.974380 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.974417 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:16.974398 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:16.974539 20029 server_base.cc:1034] running on GCE node
I20260924 03:12:16.974737 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:16.974989 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:16.976126 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790219536976113 us; error 41 us; skew 500 ppm
I20260924 03:12:16.977264 20029 webserver.cc:460] Webserver started at http://127.0.98.62:42915/ using document root <none> and password file <none>
I20260924 03:12:16.977489 20029 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:16.977555 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:16.977705 20029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:16.978672 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "72eb9db6628a406d82a0e1daf1c21512"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "10a6d13633b1e0bb9abe62d27a071711"
server_key_iv: "ccc091a6b3f7a3e7d892b98f9af456ce"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.978987 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "72eb9db6628a406d82a0e1daf1c21512"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "10a6d13633b1e0bb9abe62d27a071711"
server_key_iv: "ccc091a6b3f7a3e7d892b98f9af456ce"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.980156 20029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:16.980759 20043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:16.980952 20029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:16.981045 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "72eb9db6628a406d82a0e1daf1c21512"
format_stamp: "Formatted at 2026-09-24 03:12:16 on dist-test-slave-jtnz"
server_key: "10a6d13633b1e0bb9abe62d27a071711"
server_key_iv: "ccc091a6b3f7a3e7d892b98f9af456ce"
server_key_version: "encryptionkey@0"
I20260924 03:12:16.981123 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:17.005566 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:17.005800 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:17.005925 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:17.009413 20029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33919
I20260924 03:12:17.009442 20095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33919 every 8 connection(s)
I20260924 03:12:17.009733 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:17.010641 20096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:17.010991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20029
I20260924 03:12:17.011096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:17.011258 392 external_mini_cluster.cc:1428] Setting key 3a8cfb1c199bca91b09448f8502d3d3b
I20260924 03:12:17.013064 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: Bootstrap starting.
I20260924 03:12:17.013586 20096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:17.013849 20096 log.cc:824] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:17.014467 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: No bootstrap required, opened a new log
I20260924 03:12:17.015904 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.011727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.98.62:33919)
0924 03:12:17.011918 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.011925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.011947 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:17.012291 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.012296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.012341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.012436 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:17.012664 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.013413 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.014360 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.014726 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.014809 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.015332 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:17.015335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:17.015335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:17.015474 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:17.015499 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:17.015507 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:17.015574 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.015658 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.015726 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260924 03:12:17.016561 20096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260924 03:12:17.016669 20096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:17.016687 20096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72eb9db6628a406d82a0e1daf1c21512, State: Initialized, Role: FOLLOWER
I20260924 03:12:17.016819 20096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260924 03:12:17.016898 20096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:17.016937 20096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:17.016980 20096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:17.017443 20096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260924 03:12:17.017556 20096 leader_election.cc:302] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72eb9db6628a406d82a0e1daf1c21512; no voters:
I20260924 03:12:17.017712 20096 leader_election.cc:288] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:17.017757 20101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:17.017925 20101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [term 1 LEADER]: Becoming Leader. State: Replica: 72eb9db6628a406d82a0e1daf1c21512, State: Running, Role: LEADER
I20260924 03:12:17.017973 20096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:17.018021 20101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260924 03:12:17.018312 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72eb9db6628a406d82a0e1daf1c21512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260924 03:12:17.018311 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72eb9db6628a406d82a0e1daf1c21512. Latest consensus state: current_term: 1 leader_uuid: "72eb9db6628a406d82a0e1daf1c21512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eb9db6628a406d82a0e1daf1c21512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260924 03:12:17.018440 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:17.018446 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:17.018620 20107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:17.019094 20107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:17.020533 20107 catalog_manager.cc:1347] Generated new cluster ID: 9d04ec3015ff4c3b9a3361e018787e6a
I20260924 03:12:17.020581 20107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:17.030586 20107 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:17.031054 20107 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:17.038376 20107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: Generated new TSK 0
I20260924 03:12:17.038566 20107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:17.040751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:17.113418 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:17.113574 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:17.113590 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:17.114645 20120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:17.114682 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:17.114704 20120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:12:17.115828 20120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:17.116021 20120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:17.116302 20120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:17.121055 20126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.121094 20128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.121076 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:17.121286 20120 server_base.cc:1034] running on GCE node
I20260924 03:12:17.121459 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:17.121752 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:17.122937 20120 hybrid_clock.cc:648] HybridClock initialized: now 1790219537122925 us; error 44 us; skew 500 ppm
I20260924 03:12:17.124101 20120 webserver.cc:460] Webserver started at http://127.0.98.1:43669/ using document root <none> and password file <none>
I20260924 03:12:17.124327 20120 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:17.124374 20120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:17.124483 20120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:17.125659 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data/instance:
uuid: "551b603ea8874978926d18598266bff2"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "e2a9cc97335a64415990f11ffaef811a"
server_key_iv: "28a7afff3687c020e8ef08f7637a00d9"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.125998 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal/instance:
uuid: "551b603ea8874978926d18598266bff2"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "e2a9cc97335a64415990f11ffaef811a"
server_key_iv: "28a7afff3687c020e8ef08f7637a00d9"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.127450 20120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:17.128168 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.128369 20120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:17.128454 20120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal
uuid: "551b603ea8874978926d18598266bff2"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "e2a9cc97335a64415990f11ffaef811a"
server_key_iv: "28a7afff3687c020e8ef08f7637a00d9"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.128513 20120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:17.139430 20120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:17.139632 20120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:17.139717 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:17.139856 20120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:17.140070 20120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:17.140096 20120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.140113 20120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:17.140125 20120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.145812 20120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36643
I20260924 03:12:17.145877 20247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36643 every 8 connection(s)
I20260924 03:12:17.146145 20120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:12:17.150301 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.146766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34861 (local address 127.0.98.62:33919)
0924 03:12:17.146802 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.146804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.147005 (+ 201us) server_negotiation.cc:407] Connection header received
0924 03:12:17.147373 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.147374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.147385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.147433 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:17.147982 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.148422 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.149232 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.149393 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.149438 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.150070 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:17.150071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:17.150071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:17.150075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:17.150089 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:17.150092 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:17.150131 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.150215 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.150241 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:17.150542 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.146744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.1:34861)
0924 03:12:17.146888 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.146914 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.147299 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.147474 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.147476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.147483 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:17.147916 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.147919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.148491 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.148492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.149165 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.149167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.149455 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.149456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.149546 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.149934 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:17.149935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:17.149974 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:17.149993 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:17.150131 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:17.150133 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:17.150135 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.150225 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.150314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260924 03:12:17.151005 20248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260924 03:12:17.151082 20248 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:17.151204 20248 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260924 03:12:17.151573 20060 ts_manager.cc:194] Registered new tserver with Master: 551b603ea8874978926d18598266bff2 (127.0.98.1:36643)
I20260924 03:12:17.152143 20060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34861
I20260924 03:12:17.155109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20120
I20260924 03:12:17.155189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:12:17.155303 392 external_mini_cluster.cc:1428] Setting key c883e6bd19704e6b73badb35d0c5ab30
I20260924 03:12:17.156087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:17.228156 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:17.228322 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:17.228343 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:17.229334 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:17.229364 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:17.229385 20251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:12:17.230613 20251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:17.230769 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:17.230995 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:17.235603 20257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.235597 20256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.235598 20259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:17.235782 20251 server_base.cc:1034] running on GCE node
I20260924 03:12:17.235991 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:17.236246 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:17.237386 20251 hybrid_clock.cc:648] HybridClock initialized: now 1790219537237373 us; error 32 us; skew 500 ppm
I20260924 03:12:17.238530 20251 webserver.cc:460] Webserver started at http://127.0.98.2:35491/ using document root <none> and password file <none>
I20260924 03:12:17.238763 20251 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:17.238828 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:17.238971 20251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:17.239993 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b832cfe49cc459b97d3ef467a5074ad"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "1d58c16c22ae66b55b6bb9363749a971"
server_key_iv: "8425e7b4a5ddef4c3fda57dd453b21d5"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.240346 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b832cfe49cc459b97d3ef467a5074ad"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "1d58c16c22ae66b55b6bb9363749a971"
server_key_iv: "8425e7b4a5ddef4c3fda57dd453b21d5"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.241680 20251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:17.242394 20265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.242547 20251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.242646 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal
uuid: "8b832cfe49cc459b97d3ef467a5074ad"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "1d58c16c22ae66b55b6bb9363749a971"
server_key_iv: "8425e7b4a5ddef4c3fda57dd453b21d5"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.242718 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:17.255151 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:17.255400 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:17.255538 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:17.255748 20251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:17.256031 20251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:17.256079 20251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.256110 20251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:17.256136 20251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.261659 20251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40963
I20260924 03:12:17.261705 20378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40963 every 8 connection(s)
I20260924 03:12:17.261967 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:12:17.266065 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.262596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58693 (local address 127.0.98.62:33919)
0924 03:12:17.262672 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.262674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.262834 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:12:17.263253 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.263255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.263263 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.263311 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:17.263849 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.264292 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.265098 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.265259 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.265304 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.265873 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:17.265874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:17.265875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:17.265878 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:17.265892 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:17.265895 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:17.265932 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.265988 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.266007 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:12:17.266248 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.262563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.2:58693)
0924 03:12:17.262726 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.262755 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.263174 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.263367 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.263370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.263377 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:17.263789 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.263792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.264360 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.264361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.265032 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.265034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.265321 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.265322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.265411 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.265763 (+ 352us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:17.265763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:17.265784 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:17.265802 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:17.265932 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:17.265934 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:17.265936 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.265975 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.266036 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20260924 03:12:17.266721 20379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260924 03:12:17.266814 20379 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:17.266968 20379 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260924 03:12:17.267277 20060 ts_manager.cc:194] Registered new tserver with Master: 8b832cfe49cc459b97d3ef467a5074ad (127.0.98.2:40963)
I20260924 03:12:17.267606 20060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58693
I20260924 03:12:17.270161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20251
I20260924 03:12:17.270288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:12:17.270440 392 external_mini_cluster.cc:1428] Setting key 3772eb4608844c9f7141931c1d63835b
I20260924 03:12:17.271174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--builtin_ntp_servers=127.0.98.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:12:17.346616 20382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:17.346796 20382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:17.346827 20382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:17.347844 20382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:17.347901 20382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:17.347941 20382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:12:17.349298 20382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:17.349525 20382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:17.349766 20382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:17.354194 20387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.354277 20388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:17.354334 20390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:17.354408 20382 server_base.cc:1034] running on GCE node
I20260924 03:12:17.354671 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:17.354861 20382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:17.356022 20382 hybrid_clock.cc:648] HybridClock initialized: now 1790219537356006 us; error 30 us; skew 500 ppm
I20260924 03:12:17.357103 20382 webserver.cc:460] Webserver started at http://127.0.98.3:34911/ using document root <none> and password file <none>
I20260924 03:12:17.357316 20382 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:17.357381 20382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:17.357510 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:17.358863 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data/instance:
uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "814a2d041a5ddcc13a4d1184729e4f0a"
server_key_iv: "99cfb3b23e4f5c1b455ec02f8adb4e32"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.359208 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal/instance:
uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "814a2d041a5ddcc13a4d1184729e4f0a"
server_key_iv: "99cfb3b23e4f5c1b455ec02f8adb4e32"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.360353 20382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:17.360982 20396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.361132 20382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:17.361228 20382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal
uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7"
format_stamp: "Formatted at 2026-09-24 03:12:17 on dist-test-slave-jtnz"
server_key: "814a2d041a5ddcc13a4d1184729e4f0a"
server_key_iv: "99cfb3b23e4f5c1b455ec02f8adb4e32"
server_key_version: "encryptionkey@0"
I20260924 03:12:17.361310 20382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:17.371611 20382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:17.371871 20382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:17.372014 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:17.372205 20382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:12:17.372491 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:12:17.372539 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.372572 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:12:17.372597 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:17.377731 20382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44565
I20260924 03:12:17.377831 20509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44565 every 8 connection(s)
I20260924 03:12:17.378093 20382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:12:17.382716 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.378792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42817 (local address 127.0.98.62:33919)
0924 03:12:17.378833 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.378835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.379005 (+ 170us) server_negotiation.cc:407] Connection header received
0924 03:12:17.379445 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.379446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.379459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.379506 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:17.380094 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.380582 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.381491 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.381700 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.381774 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.382479 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:17.382481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:17.382481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:17.382486 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:17.382498 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:17.382501 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:17.382544 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.382623 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.382646 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:17.382915 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.378754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.3:42817)
0924 03:12:17.378899 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.378934 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.379354 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.379577 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.379580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.379589 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:17.380008 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.380012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.380667 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.380668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.381405 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.381407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.381780 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.381781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.381903 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.382361 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:12:17.382362 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:17.382384 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:12:17.382403 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:17.382546 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:17.382550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:17.382553 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.382626 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.382708 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260924 03:12:17.383291 20510 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260924 03:12:17.383364 20510 heartbeater.cc:464] Registering TS with master...
I20260924 03:12:17.383476 20510 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260924 03:12:17.383778 20060 ts_manager.cc:194] Registered new tserver with Master: 28fcc1b91cb1403bbc6a7be0e7c084a7 (127.0.98.3:44565)
I20260924 03:12:17.384155 20060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42817
I20260924 03:12:17.385303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20382
I20260924 03:12:17.385376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:12:17.385507 392 external_mini_cluster.cc:1428] Setting key ab60072e3077f6eb10673bae58b46520
I20260924 03:12:17.386401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:12:17.390132 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.387596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.0.98.62:33919)
0924 03:12:17.387653 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.387655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.387736 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:12:17.387790 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.387791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.387799 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.387849 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:17.388000 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.388451 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.389155 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.389346 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.389423 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.389933 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:17.389934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:17.389934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:17.389938 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:17.389948 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:17.389951 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:17.389996 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.390047 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.390062 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:12:17.391474 20060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:12:17.392927 20060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:12:17.398692 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.395090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.98.1:36643)
0924 03:12:17.395289 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.395293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.395319 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:12:17.395361 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.395363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.395378 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.395461 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:17.395626 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396646 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.397541 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.398307 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.398378 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.398457 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.398585 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.398618 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260924 03:12:17.398686 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.394906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.0.98.3:44565)
0924 03:12:17.395252 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.395256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.395266 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:17.395300 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.395301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.395315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.395386 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:17.395547 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396567 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.397730 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.398382 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.398438 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.398556 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.398581 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.398614 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":103,"threads_started":1}
I20260924 03:12:17.398743 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.394761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44565 (local address 127.0.0.1:36258)
0924 03:12:17.394877 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.394909 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.394985 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.395403 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.395406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.395408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:17.395461 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.395463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396671 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.396672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.397660 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.397662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.398452 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.398452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.398519 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.398522 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.398563 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.398631 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":55,"thread_start_us":60,"threads_started":1}
I20260924 03:12:17.398787 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.394795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36643 (local address 127.0.0.1:34026)
0924 03:12:17.394919 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.394928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.394989 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.395456 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.395457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.395458 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:17.395487 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.395489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396699 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.396700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.397363 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.397364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.398378 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.398379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.398452 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.398483 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.398563 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.398631 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260924 03:12:17.399323 20444 tablet_service.cc:1467] Processing CreateTablet for tablet eaf606e82f7e41e2803016f2a0fb0113 (DEFAULT_TABLE table=test-table [id=7900e0ceeb664f30821bcc0b7898d212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:17.399641 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf606e82f7e41e2803016f2a0fb0113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:17.399857 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.395008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40963 (local address 127.0.0.1:59434)
0924 03:12:17.395125 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.395135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.395173 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.395557 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.395558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.395560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:17.395605 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.395607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396479 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.396480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.397655 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.397658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.399637 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.399638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.399697 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.399702 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.399763 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.399798 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 03:12:17.399753 20182 tablet_service.cc:1467] Processing CreateTablet for tablet eaf606e82f7e41e2803016f2a0fb0113 (DEFAULT_TABLE table=test-table [id=7900e0ceeb664f30821bcc0b7898d212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:17.399885 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.395211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.0.98.2:40963)
0924 03:12:17.395398 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.395403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.395412 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:17.395448 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.395450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.395463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.395678 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:17.395741 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.396398 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.398806 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.399547 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.399622 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.399698 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.399772 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.399813 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260924 03:12:17.400089 20182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf606e82f7e41e2803016f2a0fb0113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:17.400475 20313 tablet_service.cc:1467] Processing CreateTablet for tablet eaf606e82f7e41e2803016f2a0fb0113 (DEFAULT_TABLE table=test-table [id=7900e0ceeb664f30821bcc0b7898d212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:12:17.400751 20313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf606e82f7e41e2803016f2a0fb0113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:17.402730 20524 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: Bootstrap starting.
I20260924 03:12:17.403268 20524 tablet_bootstrap.cc:654] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:17.403565 20524 log.cc:824] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:17.403708 20526 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Bootstrap starting.
I20260924 03:12:17.403903 20525 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Bootstrap starting.
I20260924 03:12:17.404239 20526 tablet_bootstrap.cc:654] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:17.404443 20525 tablet_bootstrap.cc:654] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:17.404552 20526 log.cc:824] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:17.404744 20525 log.cc:824] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:17.405283 20526 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: No bootstrap required, opened a new log
I20260924 03:12:17.405381 20525 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: No bootstrap required, opened a new log
I20260924 03:12:17.405391 20524 tablet_bootstrap.cc:492] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: No bootstrap required, opened a new log
I20260924 03:12:17.405390 20526 ts_tablet_manager.cc:1397] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:17.405447 20524 ts_tablet_manager.cc:1397] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:12:17.405512 20525 ts_tablet_manager.cc:1397] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:12:17.407074 20526 raft_consensus.cc:348] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.407208 20526 raft_consensus.cc:374] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:17.407264 20526 raft_consensus.cc:729] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b832cfe49cc459b97d3ef467a5074ad, State: Initialized, Role: FOLLOWER
I20260924 03:12:17.407374 20526 consensus_queue.cc:260] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.407622 20526 ts_tablet_manager.cc:1428] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:12:17.407749 20379 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260924 03:12:17.407809 20525 raft_consensus.cc:348] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.407815 20524 raft_consensus.cc:348] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.407938 20525 raft_consensus.cc:374] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:17.407938 20524 raft_consensus.cc:374] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:17.407969 20525 raft_consensus.cc:729] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fcc1b91cb1403bbc6a7be0e7c084a7, State: Initialized, Role: FOLLOWER
I20260924 03:12:17.407969 20524 raft_consensus.cc:729] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551b603ea8874978926d18598266bff2, State: Initialized, Role: FOLLOWER
I20260924 03:12:17.408092 20524 consensus_queue.cc:260] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.408113 20525 consensus_queue.cc:260] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.408351 20525 ts_tablet_manager.cc:1428] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:12:17.408351 20524 ts_tablet_manager.cc:1428] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:12:17.408383 20510 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260924 03:12:17.408461 20248 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
W20260924 03:12:17.512456 20380 tablet.cc:2369] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:17.547762 20531 raft_consensus.cc:482] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:12:17.547879 20531 raft_consensus.cc:504] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.548271 20531 leader_election.cc:288] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 551b603ea8874978926d18598266bff2 (127.0.98.1:36643), 8b832cfe49cc459b97d3ef467a5074ad (127.0.98.2:40963)
I20260924 03:12:17.550987 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.548527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57223 (local address 127.0.98.1:36643)
0924 03:12:17.548575 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.548577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.548587 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:17.548647 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.548648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.548660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.548716 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:17.548869 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.549327 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550278 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.550633 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550715 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.550724 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.550867 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.550923 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:12:17.551009 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.548450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36643 (local address 127.0.98.3:57223)
0924 03:12:17.548513 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.548536 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.548592 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.548730 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.548730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.548732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:17.548789 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.548790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.549430 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.549431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.550134 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.550135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.550713 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.550783 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.550785 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.550859 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.550918 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:12:17.551223 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.548624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55953 (local address 127.0.98.2:40963)
0924 03:12:17.548667 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.548669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.548821 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:12:17.548892 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.548893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.548901 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.548970 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:12:17.549116 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.549553 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550478 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.550822 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550923 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.550935 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.551111 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.551150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:12:17.551426 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.548587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40963 (local address 127.0.98.3:55953)
0924 03:12:17.548745 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.548754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:17.548831 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:17.548972 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:17.548973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:17.548975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:12:17.549049 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.549051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.549641 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.549642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.550397 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:17.550399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.550947 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.550948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:17.551019 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.551022 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:12:17.551305 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 03:12:17.551353 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260924 03:12:17.551381 20202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf606e82f7e41e2803016f2a0fb0113" candidate_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551b603ea8874978926d18598266bff2" is_pre_election: true
I20260924 03:12:17.551564 20202 raft_consensus.cc:2454] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fcc1b91cb1403bbc6a7be0e7c084a7 in term 0.
I20260924 03:12:17.551753 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf606e82f7e41e2803016f2a0fb0113" candidate_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b832cfe49cc459b97d3ef467a5074ad" is_pre_election: true
I20260924 03:12:17.551813 20400 leader_election.cc:302] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fcc1b91cb1403bbc6a7be0e7c084a7, 551b603ea8874978926d18598266bff2; no voters:
I20260924 03:12:17.551894 20333 raft_consensus.cc:2454] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fcc1b91cb1403bbc6a7be0e7c084a7 in term 0.
I20260924 03:12:17.551961 20531 raft_consensus.cc:2781] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:12:17.552035 20531 raft_consensus.cc:482] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:12:17.552067 20531 raft_consensus.cc:3037] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:17.552723 20531 raft_consensus.cc:504] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.552841 20531 leader_election.cc:288] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [CANDIDATE]: Term 1 election: Requested vote from peers 551b603ea8874978926d18598266bff2 (127.0.98.1:36643), 8b832cfe49cc459b97d3ef467a5074ad (127.0.98.2:40963)
I20260924 03:12:17.552982 20202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf606e82f7e41e2803016f2a0fb0113" candidate_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551b603ea8874978926d18598266bff2"
I20260924 03:12:17.552994 20333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf606e82f7e41e2803016f2a0fb0113" candidate_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b832cfe49cc459b97d3ef467a5074ad"
I20260924 03:12:17.553084 20202 raft_consensus.cc:3037] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:17.553079 20333 raft_consensus.cc:3037] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:17.553761 20202 raft_consensus.cc:2454] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fcc1b91cb1403bbc6a7be0e7c084a7 in term 1.
I20260924 03:12:17.553771 20333 raft_consensus.cc:2454] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fcc1b91cb1403bbc6a7be0e7c084a7 in term 1.
I20260924 03:12:17.553936 20400 leader_election.cc:302] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fcc1b91cb1403bbc6a7be0e7c084a7, 8b832cfe49cc459b97d3ef467a5074ad; no voters:
I20260924 03:12:17.554047 20531 raft_consensus.cc:2781] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:17.554222 20531 raft_consensus.cc:686] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 1 LEADER]: Becoming Leader. State: Replica: 28fcc1b91cb1403bbc6a7be0e7c084a7, State: Running, Role: LEADER
I20260924 03:12:17.554333 20531 consensus_queue.cc:237] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } }
I20260924 03:12:17.555083 20059 catalog_manager.cc:5495] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28fcc1b91cb1403bbc6a7be0e7c084a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fcc1b91cb1403bbc6a7be0e7c084a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 } health_report { overall_health: UNKNOWN } } }
I20260924 03:12:17.580304 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.578412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.0.98.3:44565)
0924 03:12:17.578451 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:17.578453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:17.578482 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:12:17.578533 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:17.578534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:17.578547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:17.578600 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:17.578699 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.579114 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.579736 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:17.579897 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:17.579941 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:17.580061 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:17.580144 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:17.580199 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:17.580229 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 03:12:17.580241 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:12:17.581873 20333 raft_consensus.cc:1261] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 1 FOLLOWER]: Refusing update from remote peer 28fcc1b91cb1403bbc6a7be0e7c084a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:17.581951 20202 raft_consensus.cc:1261] T eaf606e82f7e41e2803016f2a0fb0113 P 551b603ea8874978926d18598266bff2 [term 1 FOLLOWER]: Refusing update from remote peer 28fcc1b91cb1403bbc6a7be0e7c084a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:17.582207 20531 consensus_queue.cc:1035] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b832cfe49cc459b97d3ef467a5074ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:17.582329 20534 consensus_queue.cc:1035] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "551b603ea8874978926d18598266bff2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:17.584438 20537 mvcc.cc:205] Tried to move back new op lower bound from 7332739225933148160 to 7332739225823125504. Current Snapshot: MvccSnapshot[applied={T|T < 7332739225933148160}]
I20260924 03:12:17.584760 20538 mvcc.cc:205] Tried to move back new op lower bound from 7332739225933148160 to 7332739225823125504. Current Snapshot: MvccSnapshot[applied={T|T < 7332739225933148160}]
I20260924 03:12:17.585201 20539 mvcc.cc:205] Tried to move back new op lower bound from 7332739225933148160 to 7332739225823125504. Current Snapshot: MvccSnapshot[applied={T|T < 7332739225933148160}]
I20260924 03:12:17.586575 20059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:17.586663 20059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:12:17.587361 20059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72eb9db6628a406d82a0e1daf1c21512: Sending DeleteTablet for 3 replicas of tablet eaf606e82f7e41e2803016f2a0fb0113
I20260924 03:12:17.587649 20444 tablet_service.cc:1514] Processing DeleteTablet for tablet eaf606e82f7e41e2803016f2a0fb0113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:17 UTC) from {username='slave'} at 127.0.0.1:36258
I20260924 03:12:17.587862 20546 tablet_replica.cc:330] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: stopping tablet replica
I20260924 03:12:17.587884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20120
I20260924 03:12:17.588013 20546 raft_consensus.cc:2229] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:12:17.588099 20546 raft_consensus.cc:2258] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:17.588430 20313 tablet_service.cc:1514] Processing DeleteTablet for tablet eaf606e82f7e41e2803016f2a0fb0113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:12:17 UTC) from {username='slave'} at 127.0.0.1:59434
I20260924 03:12:17.588634 20547 tablet_replica.cc:330] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: stopping tablet replica
I20260924 03:12:17.588717 20547 raft_consensus.cc:2229] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:12:17.588807 20547 raft_consensus.cc:2258] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:12:17.589360 20546 ts_tablet_manager.cc:1905] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:17.590603 20547 ts_tablet_manager.cc:1905] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:12:17.592490 20547 ts_tablet_manager.cc:1918] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:17.592573 20547 log.cc:1196] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-1/wal/wals/eaf606e82f7e41e2803016f2a0fb0113
I20260924 03:12:17.592810 20547 ts_tablet_manager.cc:1939] T eaf606e82f7e41e2803016f2a0fb0113 P 8b832cfe49cc459b97d3ef467a5074ad: Deleting consensus metadata
I20260924 03:12:17.593189 20546 ts_tablet_manager.cc:1918] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:12:17.593200 20047 catalog_manager.cc:4841] TS 8b832cfe49cc459b97d3ef467a5074ad (127.0.98.2:40963): tablet eaf606e82f7e41e2803016f2a0fb0113 (table test-table [id=7900e0ceeb664f30821bcc0b7898d212]) successfully deleted
I20260924 03:12:17.593297 20546 log.cc:1196] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Deleting WAL directory at /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219310260848-392-0/minicluster-data/ts-2/wal/wals/eaf606e82f7e41e2803016f2a0fb0113
I20260924 03:12:17.593530 20546 ts_tablet_manager.cc:1939] T eaf606e82f7e41e2803016f2a0fb0113 P 28fcc1b91cb1403bbc6a7be0e7c084a7: Deleting consensus metadata
W20260924 03:12:17.593730 20047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36643 (error 108)
I20260924 03:12:17.593909 20044 catalog_manager.cc:4841] TS 28fcc1b91cb1403bbc6a7be0e7c084a7 (127.0.98.3:44565): tablet eaf606e82f7e41e2803016f2a0fb0113 (table test-table [id=7900e0ceeb664f30821bcc0b7898d212]) successfully deleted
I20260924 03:12:17.594120 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:17.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36643 (local address 127.0.0.1:34036)
0924 03:12:17.594065 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:17.594100 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 03:12:17.594251 20047 catalog_manager.cc:4584] TS 551b603ea8874978926d18598266bff2 (127.0.98.1:36643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eaf606e82f7e41e2803016f2a0fb0113: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36643: connect: Connection refused (error 111)
I20260924 03:12:17.594359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20251
I20260924 03:12:17.604846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20382
I20260924 03:12:17.608940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20029
2026-09-24T03:12:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (737 ms)
[----------] 6 tests from EncryptionPolicyTest (7913 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:17 dist-test-slave-jtnz krb5kdc[20551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:17 dist-test-slave-jtnz krb5kdc[20551](info): set up 2 sockets
Sep 24 03:12:17 dist-test-slave-jtnz krb5kdc[20551](info): commencing operation
krb5kdc: starting...
W20260924 03:12:18.589965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:18.696187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:18.766952 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:18.767160 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:18.767198 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:18.768265 20564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:18.768324 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:18.768347 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:18.768373 20564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:18.768405 20564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:18.769682 20564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:18.769946 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:18.770154 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:18.774484 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:18.774492 20570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:18.774498 20572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:18.774580 20564 server_base.cc:1034] running on GCE node
I20260924 03:12:18.774896 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:18.775117 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:18.776286 20564 hybrid_clock.cc:648] HybridClock initialized: now 1790219538776274 us; error 24 us; skew 500 ppm
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:18.778528 20564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:18.778908 20564 webserver.cc:460] Webserver started at http://127.0.0.1:42939/ using document root <none> and password file <none>
I20260924 03:12:18.779112 20564 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:18.779177 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:18.779335 20564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:18.780265 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "20e8702ef4ab4eb69a581a89e7559cbb"
format_stamp: "Formatted at 2026-09-24 03:12:18 on dist-test-slave-jtnz"
server_key: "25b4245fb2ffb8b43729a05f0eaa1218"
server_key_iv: "994ad3645b5189b8c48f34b5ce8dbc04"
server_key_version: "encryptionkey@0"
I20260924 03:12:18.780610 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "20e8702ef4ab4eb69a581a89e7559cbb"
format_stamp: "Formatted at 2026-09-24 03:12:18 on dist-test-slave-jtnz"
server_key: "25b4245fb2ffb8b43729a05f0eaa1218"
server_key_iv: "994ad3645b5189b8c48f34b5ce8dbc04"
server_key_version: "encryptionkey@0"
I20260924 03:12:18.781864 20564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:12:18.782574 20579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:18.782732 20564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:18.782825 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "20e8702ef4ab4eb69a581a89e7559cbb"
format_stamp: "Formatted at 2026-09-24 03:12:18 on dist-test-slave-jtnz"
server_key: "25b4245fb2ffb8b43729a05f0eaa1218"
server_key_iv: "994ad3645b5189b8c48f34b5ce8dbc04"
server_key_version: "encryptionkey@0"
I20260924 03:12:18.782898 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:18.795145 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:18.796613 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:18.796749 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:18.800372 20564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44779
I20260924 03:12:18.800411 20631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44779 every 8 connection(s)
I20260924 03:12:18.800694 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:18.801426 20632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:18.803473 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb: Bootstrap starting.
I20260924 03:12:18.804001 20632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:18.804272 20632 log.cc:824] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:18.804898 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb: No bootstrap required, opened a new log
I20260924 03:12:18.806402 20632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } }
I20260924 03:12:18.806520 20632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:18.806562 20632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e8702ef4ab4eb69a581a89e7559cbb, State: Initialized, Role: FOLLOWER
I20260924 03:12:18.806707 20632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } }
I20260924 03:12:18.806780 20632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:18.806820 20632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:18.806865 20632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:18.807320 20632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } }
I20260924 03:12:18.807422 20632 leader_election.cc:302] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e8702ef4ab4eb69a581a89e7559cbb; no voters:
I20260924 03:12:18.807590 20632 leader_election.cc:288] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:18.807646 20635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:18.807793 20635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [term 1 LEADER]: Becoming Leader. State: Replica: 20e8702ef4ab4eb69a581a89e7559cbb, State: Running, Role: LEADER
I20260924 03:12:18.807809 20632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:18.807904 20635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } }
I20260924 03:12:18.808251 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e8702ef4ab4eb69a581a89e7559cbb. Latest consensus state: current_term: 1 leader_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } } }
I20260924 03:12:18.808357 20636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e8702ef4ab4eb69a581a89e7559cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44779 } } }
I20260924 03:12:18.808436 20636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:18.808400 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:18.808825 20644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:18.809273 20644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:18.810065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20564
I20260924 03:12:18.810209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:18.810403 392 external_mini_cluster.cc:1428] Setting key 0f9e0e7598d5929e1d038a7524803832
I20260924 03:12:18.810953 20644 catalog_manager.cc:1347] Generated new cluster ID: 3f3f6377eb5748d3937bc41b80fb07f4
I20260924 03:12:18.811056 20644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:12:18.820572 20644 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:18.821208 20644 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:18.824025 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:18.810822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42789 (local address 127.0.0.1:44779)
0924 03:12:18.814293 (+ 3471us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:18.814301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:18.814329 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:12:18.814426 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:18.814432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:18.814478 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:18.814604 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:18.815303 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:18.816741 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:18.817714 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:18.818316 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:18.818406 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:18.820850 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:18.820853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:18.820854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:18.820863 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:18.822645 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:18.823025 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:18.823027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:18.823027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:18.823108 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:18.823312 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:18.823313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:18.823313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:18.823513 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:18.823645 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:18.823769 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 03:12:18.823846 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3431,"thread_start_us":65,"threads_started":1}
I20260924 03:12:18.824939 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:18.826668 20644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e8702ef4ab4eb69a581a89e7559cbb: Generated new TSK 0
I20260924 03:12:18.826824 20644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:18.831981 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:18.826252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42241 (local address 127.0.0.1:44779)
0924 03:12:18.826281 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:18.826282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:18.826404 (+ 122us) server_negotiation.cc:407] Connection header received
0924 03:12:18.826523 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:18.826524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:18.826532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:18.826580 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:18.827165 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:18.827624 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:18.828299 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:18.828535 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:18.828618 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:18.829648 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:18.829649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:18.829650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:18.829655 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:18.830912 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:18.831249 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:18.831250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:18.831250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:18.831305 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:18.831518 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:18.831519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:18.831519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:18.831616 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:18.831733 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:18.831840 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:12:18.831898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:18.833462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20564
W20260924 03:12:18.836995 20549 connection.cc:535] client connection to 127.0.0.1:44779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44779): Connection reset by peer (error 104)
2026-09-24T03:12:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20663](info): set up 2 sockets
Sep 24 03:12:18 dist-test-slave-jtnz krb5kdc[20663](info): commencing operation
krb5kdc: starting...
W20260924 03:12:19.781977 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:19 dist-test-slave-jtnz krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:19.880101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:19.951680 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:19.951853 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:19.951874 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:19.952880 20676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:19.952921 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:19.952934 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:19.952944 20676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:19.952986 20676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:19.954411 20676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:19.954636 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:19.954838 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:19.959127 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.959126 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:19.959178 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:19.959286 20676 server_base.cc:1034] running on GCE node
I20260924 03:12:19.959619 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:19.959878 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:19.961014 20676 hybrid_clock.cc:648] HybridClock initialized: now 1790219539961005 us; error 26 us; skew 500 ppm
Sep 24 03:12:19 dist-test-slave-jtnz krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:19.967736 20676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:19.968137 20676 webserver.cc:460] Webserver started at http://127.0.0.1:42033/ using document root <none> and password file <none>
I20260924 03:12:19.968362 20676 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:19.968420 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:19.968551 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:19.969431 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "181553c3286740ec8ed376b6458ad152"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-jtnz"
server_key: "d007ed1ff308d3056268496ac4a02868"
server_key_iv: "a9cb1870db8ad10d307ffb19812a6f0a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.969740 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "181553c3286740ec8ed376b6458ad152"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-jtnz"
server_key: "d007ed1ff308d3056268496ac4a02868"
server_key_iv: "a9cb1870db8ad10d307ffb19812a6f0a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.970959 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:19.971659 20691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:19.971810 20676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:19.971903 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "181553c3286740ec8ed376b6458ad152"
format_stamp: "Formatted at 2026-09-24 03:12:19 on dist-test-slave-jtnz"
server_key: "d007ed1ff308d3056268496ac4a02868"
server_key_iv: "a9cb1870db8ad10d307ffb19812a6f0a"
server_key_version: "encryptionkey@0"
I20260924 03:12:19.971978 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:19.983531 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:19.985016 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:19.985162 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:19.988649 20676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44711
I20260924 03:12:19.988682 20743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44711 every 8 connection(s)
I20260924 03:12:19.988978 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:19.989730 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:19.991671 20744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: Bootstrap starting.
I20260924 03:12:19.992153 20744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:19.992389 20744 log.cc:824] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:19.992925 20744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: No bootstrap required, opened a new log
I20260924 03:12:19.993747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20676
I20260924 03:12:19.993849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:19.994035 392 external_mini_cluster.cc:1428] Setting key fa2dc735d922f92f48426340ee8a0242
I20260924 03:12:19.994586 20744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 03:12:19.994769 20744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:19.994820 20744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181553c3286740ec8ed376b6458ad152, State: Initialized, Role: FOLLOWER
I20260924 03:12:19.994946 20744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 03:12:19.995030 20744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:19.995113 20744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:19.995170 20744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:19.995854 20744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 03:12:19.995950 20744 leader_election.cc:302] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181553c3286740ec8ed376b6458ad152; no voters:
I20260924 03:12:19.996109 20744 leader_election.cc:288] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:19.996167 20749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:19.996311 20744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:19.996333 20749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [term 1 LEADER]: Becoming Leader. State: Replica: 181553c3286740ec8ed376b6458ad152, State: Running, Role: LEADER
I20260924 03:12:19.996416 20749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260924 03:12:19.996734 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181553c3286740ec8ed376b6458ad152. Latest consensus state: current_term: 1 leader_uuid: "181553c3286740ec8ed376b6458ad152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260924 03:12:19.996810 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:19.996766 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181553c3286740ec8ed376b6458ad152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181553c3286740ec8ed376b6458ad152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260924 03:12:19.996960 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152 [sys.catalog]: This master's current role is: LEADER
W20260924 03:12:19.998261 20765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:19.998312 20765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:19.998350 20758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:19.998466 20758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:20.000042 20758 catalog_manager.cc:1347] Generated new cluster ID: 73ac329ec0fe4e2997b745d43d2be253
I20260924 03:12:20.000089 20758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:20.002555 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:19.994513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52137 (local address 127.0.0.1:44711)
0924 03:12:19.994685 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:19.994691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:19.994715 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:12:19.994812 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:19.994819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:19.994869 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:19.995009 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:19.995259 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.996886 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.998055 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:19.998677 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:19.998784 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:19.999475 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:19.999478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:19.999479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:19.999699 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:19.999732 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:19.999746 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.002282 (+ 2536us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.002330 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.002359 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260924 03:12:20.003671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:20.004761 20758 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:20.005189 20758 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:20.008191 20758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181553c3286740ec8ed376b6458ad152: Generated new TSK 0
I20260924 03:12:20.008374 20758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:20.011511 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:20.008067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50511 (local address 127.0.0.1:44711)
0924 03:12:20.008098 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:20.008100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:20.008679 (+ 579us) server_negotiation.cc:407] Connection header received
0924 03:12:20.008706 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:20.008707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:20.008716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:20.008777 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:20.008879 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.009618 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.010356 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:20.010645 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:20.010708 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:20.011286 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:20.011287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:20.011287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:20.011291 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:20.011305 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:20.011309 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:20.011381 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:20.011435 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:12:20.011445 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:20.013058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20676
W20260924 03:12:20.016635 20661 connection.cc:535] client connection to 127.0.0.1:44711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44711): Connection reset by peer (error 104)
2026-09-24T03:12:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:20 dist-test-slave-jtnz krb5kdc[20775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:20 dist-test-slave-jtnz krb5kdc[20775](info): set up 2 sockets
Sep 24 03:12:20 dist-test-slave-jtnz krb5kdc[20775](info): commencing operation
krb5kdc: starting...
W20260924 03:12:20.957659 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:21 dist-test-slave-jtnz krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:21.056445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:21.129654 20788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:21.129918 20788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:21.129952 20788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:21.132052 20788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:21.132117 20788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:21.132144 20788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:21.132167 20788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:21.132198 20788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:21.134919 20788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:21.135300 20788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:21.135552 20788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:21.140642 20793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:21.140806 20796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:21.140655 20794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:21.140699 20788 server_base.cc:1034] running on GCE node
I20260924 03:12:21.141520 20788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:21.141866 20788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:21.143030 20788 hybrid_clock.cc:648] HybridClock initialized: now 1790219541143010 us; error 36 us; skew 500 ppm
Sep 24 03:12:21 dist-test-slave-jtnz krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:21.145694 20788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:21.146224 20788 webserver.cc:460] Webserver started at http://127.0.0.1:46507/ using document root <none> and password file <none>
I20260924 03:12:21.146502 20788 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:21.146590 20788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:21.146731 20788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:21.148111 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "a29a6371095246b9b7b75751296b36ca"
format_stamp: "Formatted at 2026-09-24 03:12:21 on dist-test-slave-jtnz"
server_key: "416ce4099b1d28a7ce02293d74c0968a"
server_key_iv: "4fa0c67fbbf98bc0f658e215b2264a65"
server_key_version: "encryptionkey@0"
I20260924 03:12:21.148485 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "a29a6371095246b9b7b75751296b36ca"
format_stamp: "Formatted at 2026-09-24 03:12:21 on dist-test-slave-jtnz"
server_key: "416ce4099b1d28a7ce02293d74c0968a"
server_key_iv: "4fa0c67fbbf98bc0f658e215b2264a65"
server_key_version: "encryptionkey@0"
I20260924 03:12:21.150236 20788 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 03:12:21.151111 20803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:21.151283 20788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:21.151382 20788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "a29a6371095246b9b7b75751296b36ca"
format_stamp: "Formatted at 2026-09-24 03:12:21 on dist-test-slave-jtnz"
server_key: "416ce4099b1d28a7ce02293d74c0968a"
server_key_iv: "4fa0c67fbbf98bc0f658e215b2264a65"
server_key_version: "encryptionkey@0"
I20260924 03:12:21.151464 20788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:21.155680 20788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:21.157959 20788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:21.158090 20788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:21.162335 20788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34283
I20260924 03:12:21.162387 20855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34283 every 8 connection(s)
I20260924 03:12:21.162685 20788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:21.163802 20856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:21.166898 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca: Bootstrap starting.
I20260924 03:12:21.167563 20856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:21.167877 20856 log.cc:824] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:21.168583 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca: No bootstrap required, opened a new log
I20260924 03:12:21.170001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20788
I20260924 03:12:21.170123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:21.170316 392 external_mini_cluster.cc:1428] Setting key 6b46ce23b137028de42803175eeabca0
I20260924 03:12:21.171116 20856 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260924 03:12:21.171309 20856 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:21.171341 20856 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a6371095246b9b7b75751296b36ca, State: Initialized, Role: FOLLOWER
I20260924 03:12:21.171458 20856 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260924 03:12:21.171830 20856 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:21.171883 20856 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:21.171932 20856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:21.172254 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:21.170831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60225 (local address 127.0.0.1:34283)
0924 03:12:21.170975 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:21.170982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:21.171009 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:12:21.171086 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:21.171090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:21.171108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:21.171226 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:21.171838 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:21.171841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:21.171842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:21.171846 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:21.171888 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:21.171901 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:21.171983 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:21.172051 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:12:21.172089 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260924 03:12:21.172622 20856 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260924 03:12:21.172730 20856 leader_election.cc:302] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29a6371095246b9b7b75751296b36ca; no voters:
I20260924 03:12:21.172857 20856 leader_election.cc:288] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:21.172892 20861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:21.173010 20861 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [term 1 LEADER]: Becoming Leader. State: Replica: a29a6371095246b9b7b75751296b36ca, State: Running, Role: LEADER
I20260924 03:12:21.173063 20856 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:21.173082 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:21.173094 20861 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260924 03:12:21.173403 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29a6371095246b9b7b75751296b36ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } } }
I20260924 03:12:21.173482 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:21.173588 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29a6371095246b9b7b75751296b36ca. Latest consensus state: current_term: 1 leader_uuid: "a29a6371095246b9b7b75751296b36ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29a6371095246b9b7b75751296b36ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } } }
I20260924 03:12:21.173641 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:21.173735 20868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:21.174225 20868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:21.175520 20868 catalog_manager.cc:1347] Generated new cluster ID: cda9aaaf8809476db465a7b94880c04d
I20260924 03:12:21.175575 20868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:21.179414 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:21.178444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43545 (local address 127.0.0.1:34283)
0924 03:12:21.178479 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:21.178481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:21.178611 (+ 130us) server_negotiation.cc:407] Connection header received
0924 03:12:21.178663 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:21.178664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:21.178669 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:21.178717 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:21.179225 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:21.179226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:21.179226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:21.179230 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:21.179242 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:21.179245 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:21.179305 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:21.179353 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 03:12:21.179363 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:12:21.189642 20868 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:21.189800 20868 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:21.193249 20868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29a6371095246b9b7b75751296b36ca: Generated new TSK 0
I20260924 03:12:21.193403 20868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:21.198678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20788
2026-09-24T03:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:21 dist-test-slave-jtnz krb5kdc[20887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:21 dist-test-slave-jtnz krb5kdc[20887](info): set up 2 sockets
Sep 24 03:12:21 dist-test-slave-jtnz krb5kdc[20887](info): commencing operation
krb5kdc: starting...
W20260924 03:12:22.145743 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:22.245970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:22.318488 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:22.318652 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:22.318672 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:22.319730 20900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:22.319770 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:22.319783 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:22.319828 20900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:22.319845 20900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:22.321177 20900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:22.321403 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:22.321607 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:22.325938 20900 server_base.cc:1034] running on GCE node
W20260924 03:12:22.325911 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:22.325917 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:22.325922 20908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:22.326349 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:22.326613 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:22.327792 20900 hybrid_clock.cc:648] HybridClock initialized: now 1790219542327767 us; error 57 us; skew 500 ppm
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:22.330057 20900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:22.330469 20900 webserver.cc:460] Webserver started at http://127.0.0.1:44105/ using document root <none> and password file <none>
I20260924 03:12:22.330657 20900 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:22.330713 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:22.330822 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:22.331702 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "df2b09be936247a2a8c4e31b2373839f"
format_stamp: "Formatted at 2026-09-24 03:12:22 on dist-test-slave-jtnz"
server_key: "8014c5667c8e7c320944182d41bef055"
server_key_iv: "4418dc905585b38862cbec88219125eb"
server_key_version: "encryptionkey@0"
I20260924 03:12:22.332036 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "df2b09be936247a2a8c4e31b2373839f"
format_stamp: "Formatted at 2026-09-24 03:12:22 on dist-test-slave-jtnz"
server_key: "8014c5667c8e7c320944182d41bef055"
server_key_iv: "4418dc905585b38862cbec88219125eb"
server_key_version: "encryptionkey@0"
I20260924 03:12:22.333313 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:22.333926 20915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:22.334085 20900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:22.334172 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "df2b09be936247a2a8c4e31b2373839f"
format_stamp: "Formatted at 2026-09-24 03:12:22 on dist-test-slave-jtnz"
server_key: "8014c5667c8e7c320944182d41bef055"
server_key_iv: "4418dc905585b38862cbec88219125eb"
server_key_version: "encryptionkey@0"
I20260924 03:12:22.334270 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:22.354563 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:22.356123 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:22.356276 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:22.359962 20900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41709
I20260924 03:12:22.359990 20967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41709 every 8 connection(s)
I20260924 03:12:22.360296 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:22.361052 20968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:22.363091 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f: Bootstrap starting.
I20260924 03:12:22.363574 20968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:22.363782 20968 log.cc:824] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:22.364351 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f: No bootstrap required, opened a new log
I20260924 03:12:22.365746 20968 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260924 03:12:22.365864 20968 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:22.365885 20968 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2b09be936247a2a8c4e31b2373839f, State: Initialized, Role: FOLLOWER
I20260924 03:12:22.366009 20968 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260924 03:12:22.366097 20968 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:22.366118 20968 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:22.366144 20968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:22.366637 20968 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260924 03:12:22.366715 20968 leader_election.cc:302] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2b09be936247a2a8c4e31b2373839f; no voters:
I20260924 03:12:22.366839 20968 leader_election.cc:288] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:22.366901 20971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:22.367062 20971 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [term 1 LEADER]: Becoming Leader. State: Replica: df2b09be936247a2a8c4e31b2373839f, State: Running, Role: LEADER
I20260924 03:12:22.367089 20968 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:22.367148 20971 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } }
I20260924 03:12:22.367496 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2b09be936247a2a8c4e31b2373839f. Latest consensus state: current_term: 1 leader_uuid: "df2b09be936247a2a8c4e31b2373839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } } }
I20260924 03:12:22.367611 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2b09be936247a2a8c4e31b2373839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2b09be936247a2a8c4e31b2373839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41709 } } }
I20260924 03:12:22.367638 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:22.367722 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:22.368131 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:22.368623 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:22.369663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 20900
I20260924 03:12:22.369766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:22.369899 20980 catalog_manager.cc:1347] Generated new cluster ID: b054c710d2d94ef582fb6f4c6741199d
I20260924 03:12:22.369931 392 external_mini_cluster.cc:1428] Setting key aa3eef4c56a45618236e32076b94da7f
I20260924 03:12:22.369947 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:22.379244 20980 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:22.379904 20980 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:12:22.389305 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:22.370318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45671 (local address 127.0.0.1:41709)
0924 03:12:22.370448 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:22.370456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:22.379188 (+ 8732us) server_negotiation.cc:407] Connection header received
0924 03:12:22.379327 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:22.379334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:22.379382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:22.379511 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:22.380202 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:22.381564 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:22.382620 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:22.383171 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:22.383285 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:22.385733 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:22.385737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:22.385738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:22.385748 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:22.387521 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:22.387884 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:22.387886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:22.387887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:22.387976 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:22.388160 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:22.388160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:22.388161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:22.388706 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:22.388871 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:22.388966 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:12:22.389119 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
I20260924 03:12:22.389390 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2b09be936247a2a8c4e31b2373839f: Generated new TSK 0
I20260924 03:12:22.389503 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:22.390107 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:22.397015 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:22.391456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45327 (local address 127.0.0.1:41709)
0924 03:12:22.391490 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:22.391492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:22.391584 (+ 92us) server_negotiation.cc:407] Connection header received
0924 03:12:22.391634 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:22.391635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:22.391649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:22.391690 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:22.392214 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:22.392632 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:22.393476 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:22.393649 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:22.393686 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:22.394871 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:22.394872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:22.394872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:22.394877 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:22.396117 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:22.396467 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:22.396468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:22.396468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:22.396508 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:22.396672 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:22.396673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:22.396673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:22.396748 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:22.396814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:22.396924 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 03:12:22.396969 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:12:22.398669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 20900
2026-09-24T03:12:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20999](info): set up 2 sockets
Sep 24 03:12:22 dist-test-slave-jtnz krb5kdc[20999](info): commencing operation
krb5kdc: starting...
W20260924 03:12:23.354045 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:23 dist-test-slave-jtnz krb5kdc[20999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:23.454608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:23.528566 21012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:23.528723 21012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:23.528743 21012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:23.529837 21012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:23.529876 21012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:23.529889 21012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:23.529938 21012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:23.529964 21012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:23.531410 21012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:23.531653 21012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:23.531868 21012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:23.536185 21018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:23.536163 21020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:23.536204 21012 server_base.cc:1034] running on GCE node
W20260924 03:12:23.536180 21017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:23.536512 21012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:23.536782 21012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:23.537923 21012 hybrid_clock.cc:648] HybridClock initialized: now 1790219543537912 us; error 26 us; skew 500 ppm
Sep 24 03:12:23 dist-test-slave-jtnz krb5kdc[20999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:23.540228 21012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:23.540609 21012 webserver.cc:460] Webserver started at http://127.0.0.1:41147/ using document root <none> and password file <none>
I20260924 03:12:23.540789 21012 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:23.540845 21012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:23.541000 21012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:23.541904 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "9d12d1a5f017409dbef7e4b5fae9125a"
format_stamp: "Formatted at 2026-09-24 03:12:23 on dist-test-slave-jtnz"
server_key: "0a599ea5e210e9833a7312c36592ef4e"
server_key_iv: "a1397f202a63070a4fc96421250d219a"
server_key_version: "encryptionkey@0"
I20260924 03:12:23.542287 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d12d1a5f017409dbef7e4b5fae9125a"
format_stamp: "Formatted at 2026-09-24 03:12:23 on dist-test-slave-jtnz"
server_key: "0a599ea5e210e9833a7312c36592ef4e"
server_key_iv: "a1397f202a63070a4fc96421250d219a"
server_key_version: "encryptionkey@0"
I20260924 03:12:23.543408 21012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:23.544044 21027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:23.544196 21012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:23.544282 21012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9d12d1a5f017409dbef7e4b5fae9125a"
format_stamp: "Formatted at 2026-09-24 03:12:23 on dist-test-slave-jtnz"
server_key: "0a599ea5e210e9833a7312c36592ef4e"
server_key_iv: "a1397f202a63070a4fc96421250d219a"
server_key_version: "encryptionkey@0"
I20260924 03:12:23.544358 21012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:23.550680 21012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:23.552786 21012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:23.552929 21012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:23.556722 21012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37271
I20260924 03:12:23.556747 21079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37271 every 8 connection(s)
I20260924 03:12:23.557024 21012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:23.557775 21080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:23.558115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21012
I20260924 03:12:23.558264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:23.558442 392 external_mini_cluster.cc:1428] Setting key 2073b48fc83ac3a9105938e94fb8c564
I20260924 03:12:23.560326 21080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a: Bootstrap starting.
I20260924 03:12:23.560925 21080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:23.561262 21080 log.cc:824] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:23.561950 21080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a: No bootstrap required, opened a new log
I20260924 03:12:23.563681 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:23.558927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36145 (local address 127.0.0.1:37271)
0924 03:12:23.559087 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:23.559095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:23.559128 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:12:23.559476 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:23.559481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:23.559521 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:23.559631 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:23.559866 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:23.560753 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:23.561818 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:23.562294 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:23.562369 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:23.562927 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:23.562930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:23.562931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:23.563071 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:23.563096 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:23.563106 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:23.563153 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:23.563246 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:12:23.563287 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260924 03:12:23.563647 21080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260924 03:12:23.563746 21080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:23.563799 21080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d12d1a5f017409dbef7e4b5fae9125a, State: Initialized, Role: FOLLOWER
I20260924 03:12:23.563910 21080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260924 03:12:23.564021 21080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:23.564050 21080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:23.564102 21080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:23.564690 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:23.564728 21080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260924 03:12:23.564811 21080 leader_election.cc:302] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d12d1a5f017409dbef7e4b5fae9125a; no voters:
I20260924 03:12:23.564950 21080 leader_election.cc:288] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:23.565013 21085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:23.565176 21080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:23.565217 21085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [term 1 LEADER]: Becoming Leader. State: Replica: 9d12d1a5f017409dbef7e4b5fae9125a, State: Running, Role: LEADER
I20260924 03:12:23.565374 21085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260924 03:12:23.565696 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260924 03:12:23.565765 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:23.565912 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d12d1a5f017409dbef7e4b5fae9125a. Latest consensus state: current_term: 1 leader_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d12d1a5f017409dbef7e4b5fae9125a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260924 03:12:23.565958 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:23.566552 21098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:23.567134 21098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:23.568403 21098 catalog_manager.cc:1347] Generated new cluster ID: ac4c76487b6646c896fa16a34425f4f2
I20260924 03:12:23.568471 21098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:23.569464 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:23.566096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49997 (local address 127.0.0.1:37271)
0924 03:12:23.566123 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:23.566124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:23.566374 (+ 250us) server_negotiation.cc:407] Connection header received
0924 03:12:23.566437 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:23.566437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:23.566447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:23.566493 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:23.566692 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:23.567363 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:23.568297 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:23.568580 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:23.568619 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:23.569257 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:23.569258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:23.569258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:23.569262 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:23.569275 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:23.569278 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:23.569328 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:23.569382 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:12:23.569400 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 03:12:23.574319 21098 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:23.574959 21098 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:23.589403 21098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d12d1a5f017409dbef7e4b5fae9125a: Generated new TSK 0
I20260924 03:12:23.589553 21098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:23.594161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21012
2026-09-24T03:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:23 dist-test-slave-jtnz krb5kdc[21111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:23 dist-test-slave-jtnz krb5kdc[21111](info): set up 2 sockets
Sep 24 03:12:23 dist-test-slave-jtnz krb5kdc[21111](info): commencing operation
krb5kdc: starting...
W20260924 03:12:24.561836 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:24 dist-test-slave-jtnz krb5kdc[21111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:24.660840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:24.734109 21124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:24.734292 21124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:24.734313 21124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:24.735388 21124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:24.735426 21124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:24.735440 21124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:24.735477 21124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:24.735510 21124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:24.736863 21124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:24.737022 21124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:24.737227 21124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:24.741611 21132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:24.741600 21130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:24.741636 21124 server_base.cc:1034] running on GCE node
W20260924 03:12:24.741611 21129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:24.742009 21124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:24.742322 21124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:24.743455 21124 hybrid_clock.cc:648] HybridClock initialized: now 1790219544743446 us; error 26 us; skew 500 ppm
Sep 24 03:12:24 dist-test-slave-jtnz krb5kdc[21111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:24.745527 21124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:24.745916 21124 webserver.cc:460] Webserver started at http://127.0.0.1:36723/ using document root <none> and password file <none>
I20260924 03:12:24.746117 21124 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:24.746181 21124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:24.746338 21124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:24.747186 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "e4d24a6cb27646b18a91fde3579ab5cd"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-jtnz"
server_key: "a9133f0210305c789df659c9dbf9ca21"
server_key_iv: "0af0e0e1f34e27d7e03cb04091aaf107"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.747493 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4d24a6cb27646b18a91fde3579ab5cd"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-jtnz"
server_key: "a9133f0210305c789df659c9dbf9ca21"
server_key_iv: "0af0e0e1f34e27d7e03cb04091aaf107"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.748620 21124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:24.749243 21139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:24.749411 21124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:24.749534 21124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "e4d24a6cb27646b18a91fde3579ab5cd"
format_stamp: "Formatted at 2026-09-24 03:12:24 on dist-test-slave-jtnz"
server_key: "a9133f0210305c789df659c9dbf9ca21"
server_key_iv: "0af0e0e1f34e27d7e03cb04091aaf107"
server_key_version: "encryptionkey@0"
I20260924 03:12:24.749604 21124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:24.752616 21124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:24.754298 21124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:24.754436 21124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:24.757997 21124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37459
I20260924 03:12:24.758028 21191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37459 every 8 connection(s)
I20260924 03:12:24.758340 21124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:24.759193 21192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:24.761054 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd: Bootstrap starting.
I20260924 03:12:24.761511 21192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:24.761760 21192 log.cc:824] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:24.762384 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd: No bootstrap required, opened a new log
I20260924 03:12:24.763744 21192 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260924 03:12:24.763857 21192 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:24.763876 21192 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d24a6cb27646b18a91fde3579ab5cd, State: Initialized, Role: FOLLOWER
I20260924 03:12:24.763990 21192 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260924 03:12:24.764070 21192 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:24.764091 21192 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:24.764117 21192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:24.764353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21124
I20260924 03:12:24.764472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:24.764592 21192 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260924 03:12:24.764633 392 external_mini_cluster.cc:1428] Setting key 833915283a1a7652b7dc73e3f1d3e00b
I20260924 03:12:24.764685 21192 leader_election.cc:302] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4d24a6cb27646b18a91fde3579ab5cd; no voters:
I20260924 03:12:24.764845 21192 leader_election.cc:288] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:24.764873 21195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:24.764979 21195 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [term 1 LEADER]: Becoming Leader. State: Replica: e4d24a6cb27646b18a91fde3579ab5cd, State: Running, Role: LEADER
I20260924 03:12:24.765085 21192 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:24.765118 21195 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260924 03:12:24.765475 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4d24a6cb27646b18a91fde3579ab5cd. Latest consensus state: current_term: 1 leader_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20260924 03:12:24.765480 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d24a6cb27646b18a91fde3579ab5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20260924 03:12:24.765619 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:24.765606 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:24.766001 21202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:24.766549 21202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:24.768057 21202 catalog_manager.cc:1347] Generated new cluster ID: 4d6fe183d94848a7a90881124cf6c87f
I20260924 03:12:24.768110 21202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:24.768558 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:24.765188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34907 (local address 127.0.0.1:37459)
0924 03:12:24.765339 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:24.765364 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:24.766962 (+ 1598us) server_negotiation.cc:407] Connection header received
0924 03:12:24.767197 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:24.767204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:24.767234 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:24.767370 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:24.768140 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:24.768143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:24.768144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:24.768147 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:24.768182 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:24.768195 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:24.768259 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:24.768364 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:12:24.768404 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 03:12:24.769340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:24.775032 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:24.770698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53509 (local address 127.0.0.1:37459)
0924 03:12:24.770729 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:24.770731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:24.774313 (+ 3582us) server_negotiation.cc:407] Connection header received
0924 03:12:24.774348 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:24.774349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:24.774353 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:24.774412 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:24.774854 (+ 442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:24.774855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:24.774855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:24.774859 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:24.774870 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:24.774874 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:24.774929 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:24.774963 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 03:12:24.774974 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:24.789687 21202 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:24.789826 21202 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:24.798864 21202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4d24a6cb27646b18a91fde3579ab5cd: Generated new TSK 0
I20260924 03:12:24.799017 21202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:24.803149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21124
2026-09-24T03:12:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:24 dist-test-slave-jtnz krb5kdc[21223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:24 dist-test-slave-jtnz krb5kdc[21223](info): set up 2 sockets
Sep 24 03:12:24 dist-test-slave-jtnz krb5kdc[21223](info): commencing operation
krb5kdc: starting...
W20260924 03:12:25.753893 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:25 dist-test-slave-jtnz krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:25.854907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:25.928803 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:25.929034 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:25.929068 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:25.930126 21236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:25.930190 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:25.930239 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:25.930261 21236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:25.930292 21236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:25.931643 21236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:25.931895 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:25.932111 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:25.936782 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:25.936779 21241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:25.936847 21236 server_base.cc:1034] running on GCE node
W20260924 03:12:25.936780 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:25.937224 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:25.937476 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:25.938624 21236 hybrid_clock.cc:648] HybridClock initialized: now 1790219545938607 us; error 28 us; skew 500 ppm
Sep 24 03:12:25 dist-test-slave-jtnz krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:25.946141 21236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:25.946570 21236 webserver.cc:460] Webserver started at http://127.0.0.1:42491/ using document root <none> and password file <none>
I20260924 03:12:25.946756 21236 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:25.946813 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:25.946923 21236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:25.947786 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "a457e82679d94de5bb4bb01aa8b02302"
format_stamp: "Formatted at 2026-09-24 03:12:25 on dist-test-slave-jtnz"
server_key: "6d989bca34b9f5b54f39baaa05d57580"
server_key_iv: "cea81a47abfe0f11ae4a85e3d416348d"
server_key_version: "encryptionkey@0"
I20260924 03:12:25.948105 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "a457e82679d94de5bb4bb01aa8b02302"
format_stamp: "Formatted at 2026-09-24 03:12:25 on dist-test-slave-jtnz"
server_key: "6d989bca34b9f5b54f39baaa05d57580"
server_key_iv: "cea81a47abfe0f11ae4a85e3d416348d"
server_key_version: "encryptionkey@0"
I20260924 03:12:25.949339 21236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:25.950011 21251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:25.950170 21236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:25.950285 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "a457e82679d94de5bb4bb01aa8b02302"
format_stamp: "Formatted at 2026-09-24 03:12:25 on dist-test-slave-jtnz"
server_key: "6d989bca34b9f5b54f39baaa05d57580"
server_key_iv: "cea81a47abfe0f11ae4a85e3d416348d"
server_key_version: "encryptionkey@0"
I20260924 03:12:25.950368 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:25.961059 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:25.962724 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:25.962860 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:25.966153 21236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46161
I20260924 03:12:25.966230 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46161 every 8 connection(s)
I20260924 03:12:25.966529 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:25.967289 21304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:25.968626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21236
I20260924 03:12:25.968743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:25.968922 392 external_mini_cluster.cc:1428] Setting key 47b2b1e01e93df9f651390802fff5faa
I20260924 03:12:25.969573 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302: Bootstrap starting.
I20260924 03:12:25.970118 21304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:25.970465 21304 log.cc:824] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:25.971141 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302: No bootstrap required, opened a new log
I20260924 03:12:25.973238 21304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } }
I20260924 03:12:25.973416 21304 raft_consensus.cc:374] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:25.973456 21304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a457e82679d94de5bb4bb01aa8b02302, State: Initialized, Role: FOLLOWER
I20260924 03:12:25.973593 21304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } }
I20260924 03:12:25.973685 21304 raft_consensus.cc:388] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:25.973726 21304 raft_consensus.cc:482] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:25.973755 21304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:25.974447 21304 raft_consensus.cc:504] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } }
I20260924 03:12:25.974570 21304 leader_election.cc:302] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a457e82679d94de5bb4bb01aa8b02302; no voters:
I20260924 03:12:25.974712 21304 leader_election.cc:288] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:25.974766 21309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:25.974898 21309 raft_consensus.cc:686] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [term 1 LEADER]: Becoming Leader. State: Replica: a457e82679d94de5bb4bb01aa8b02302, State: Running, Role: LEADER
I20260924 03:12:25.974946 21304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:25.975035 21309 consensus_queue.cc:237] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } }
I20260924 03:12:25.975423 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a457e82679d94de5bb4bb01aa8b02302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } } }
Sep 24 03:12:25 dist-test-slave-jtnz krb5kdc[21223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:12:25.975495 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:25.975611 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a457e82679d94de5bb4bb01aa8b02302. Latest consensus state: current_term: 1 leader_uuid: "a457e82679d94de5bb4bb01aa8b02302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a457e82679d94de5bb4bb01aa8b02302" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46161 } } }
I20260924 03:12:25.975690 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:25.976402 21316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:25.977101 21316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:25.978626 21316 catalog_manager.cc:1347] Generated new cluster ID: 96ebfc4549c6439f8e6c5286ba891a48
I20260924 03:12:25.978677 21316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:25.979609 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:25.969384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:54779 (local address 127.0.0.1:46161)
0924 03:12:25.969554 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:25.969561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:25.969583 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:12:25.969972 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:25.969978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:25.970022 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:25.970125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:25.970891 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:25.971742 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:25.972864 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:25.973227 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:25.973297 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:25.976051 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:25.976055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:25.976055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:25.976065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:25.977943 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:25.978426 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:25.978429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:25.978429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:25.978517 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:25.978791 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:25.978792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:25.978792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:25.979015 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:25.979189 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:25.979315 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:12:25.979410 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260924 03:12:25.983157 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:25.985189 21316 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:25.985791 21316 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:25.989657 21316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a457e82679d94de5bb4bb01aa8b02302: Generated new TSK 0
I20260924 03:12:25.989792 21316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:25.991473 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:25.984391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:60355 (local address 127.0.0.1:46161)
0924 03:12:25.984423 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:25.984425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:25.984980 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:12:25.985034 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:25.985035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:25.985044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:25.985105 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:25.986663 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:25.987096 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:25.987715 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:25.987888 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:25.987963 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:25.988962 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:25.988964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:25.988964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:25.988970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:25.990417 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:25.990735 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:25.990736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:25.990736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:25.990786 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:25.991007 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:25.991008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:25.991008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:25.991111 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:25.991207 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:25.991325 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:12:25.991379 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:25.993165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21236
W20260924 03:12:25.997291 21221 connection.cc:535] client connection to 127.0.0.1:46161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46161): Connection reset by peer (error 104)
2026-09-24T03:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:26 dist-test-slave-jtnz krb5kdc[21335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:26 dist-test-slave-jtnz krb5kdc[21335](info): set up 2 sockets
Sep 24 03:12:26 dist-test-slave-jtnz krb5kdc[21335](info): commencing operation
krb5kdc: starting...
W20260924 03:12:26.961926 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:27 dist-test-slave-jtnz krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:27.062374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:27.135600 21348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:27.135769 21348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:27.135790 21348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:27.136876 21348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:27.136916 21348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:27.136929 21348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:27.136965 21348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:27.136992 21348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:27.138365 21348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:27.138584 21348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:27.138797 21348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:27.143184 21353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:27.143203 21354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:27.143222 21356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:27.143245 21348 server_base.cc:1034] running on GCE node
I20260924 03:12:27.143610 21348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:27.143850 21348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:27.144982 21348 hybrid_clock.cc:648] HybridClock initialized: now 1790219547144974 us; error 26 us; skew 500 ppm
Sep 24 03:12:27 dist-test-slave-jtnz krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:27.147167 21348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:27.147596 21348 webserver.cc:460] Webserver started at http://127.0.0.1:34841/ using document root <none> and password file <none>
I20260924 03:12:27.147797 21348 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:27.147858 21348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:27.147998 21348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:27.148913 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "f69ddf6eaebc4225b1512443bdf65e42"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-jtnz"
server_key: "b6e25ed04d3792a6fa496b292fcb972a"
server_key_iv: "7b35441425c4d9c76255b4ffa65b69e2"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.149250 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "f69ddf6eaebc4225b1512443bdf65e42"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-jtnz"
server_key: "b6e25ed04d3792a6fa496b292fcb972a"
server_key_iv: "7b35441425c4d9c76255b4ffa65b69e2"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.150527 21348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:27.151160 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:27.151311 21348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:27.151397 21348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "f69ddf6eaebc4225b1512443bdf65e42"
format_stamp: "Formatted at 2026-09-24 03:12:27 on dist-test-slave-jtnz"
server_key: "b6e25ed04d3792a6fa496b292fcb972a"
server_key_iv: "7b35441425c4d9c76255b4ffa65b69e2"
server_key_version: "encryptionkey@0"
I20260924 03:12:27.151468 21348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:27.182386 21348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:27.184098 21348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:27.184249 21348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:27.188133 21348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34407
I20260924 03:12:27.188167 21415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34407 every 8 connection(s)
I20260924 03:12:27.188501 21348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:27.189275 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:27.191354 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42: Bootstrap starting.
I20260924 03:12:27.191810 21416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:27.192051 21416 log.cc:824] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:27.192646 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42: No bootstrap required, opened a new log
I20260924 03:12:27.194078 21416 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 03:12:27.194248 21416 raft_consensus.cc:374] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:27.194270 21416 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69ddf6eaebc4225b1512443bdf65e42, State: Initialized, Role: FOLLOWER
I20260924 03:12:27.194362 21416 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 03:12:27.194409 21416 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:27.194428 21416 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:27.194458 21416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:27.194887 21416 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 03:12:27.194963 21416 leader_election.cc:302] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69ddf6eaebc4225b1512443bdf65e42; no voters:
I20260924 03:12:27.195080 21416 leader_election.cc:288] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:27.195117 21419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:27.195254 21419 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [term 1 LEADER]: Becoming Leader. State: Replica: f69ddf6eaebc4225b1512443bdf65e42, State: Running, Role: LEADER
I20260924 03:12:27.195334 21416 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:27.195379 21419 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 03:12:27.195761 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f69ddf6eaebc4225b1512443bdf65e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 03:12:27.195858 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:27.195762 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69ddf6eaebc4225b1512443bdf65e42. Latest consensus state: current_term: 1 leader_uuid: "f69ddf6eaebc4225b1512443bdf65e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69ddf6eaebc4225b1512443bdf65e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 03:12:27.195983 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:27.196120 21423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:27.196349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21348
I20260924 03:12:27.196470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:27.196657 392 external_mini_cluster.cc:1428] Setting key 9cc874fa671db88cd063410305e1bd00
I20260924 03:12:27.197255 21423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:27.199323 21423 catalog_manager.cc:1347] Generated new cluster ID: 4562cc3ded3a4c56ba3384fc4681979d
I20260924 03:12:27.199375 21423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:27.202389 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.197105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:48055 (local address 127.0.0.1:34407)
0924 03:12:27.197261 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:27.197270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:27.197402 (+ 132us) server_negotiation.cc:407] Connection header received
0924 03:12:27.197534 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:27.197540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:27.197590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:27.197737 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:27.197968 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.199178 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.200189 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.200765 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.200871 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.201545 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:27.201549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:27.201549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:27.201757 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:27.201793 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:27.201806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:27.201919 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:27.202127 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:12:27.202157 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 03:12:27.203299 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:27.209403 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:27.204902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:34125 (local address 127.0.0.1:34407)
0924 03:12:27.204936 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:27.204938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:27.206318 (+ 1380us) server_negotiation.cc:407] Connection header received
0924 03:12:27.206347 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:27.206348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:27.206359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:27.206426 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:27.206517 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.207233 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.208130 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:27.208443 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:27.208505 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:27.209145 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:27.209146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:27.209146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:27.209151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:27.209164 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:27.209169 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:27.209239 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:27.209322 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:12:27.209334 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:27.221182 21423 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:27.221807 21423 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:27.236675 21423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69ddf6eaebc4225b1512443bdf65e42: Generated new TSK 0
I20260924 03:12:27.236814 21423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:27.239305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21348
W20260924 03:12:27.243649 21333 connection.cc:535] client connection to 127.0.0.1:34407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34407): Connection reset by peer (error 104)
2026-09-24T03:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:27 dist-test-slave-jtnz krb5kdc[21447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:27 dist-test-slave-jtnz krb5kdc[21447](info): set up 2 sockets
Sep 24 03:12:27 dist-test-slave-jtnz krb5kdc[21447](info): commencing operation
krb5kdc: starting...
W20260924 03:12:28.205912 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:28 dist-test-slave-jtnz krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:28.313849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:28.394910 21460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:28.395103 21460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:28.395126 21460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:28.396258 21460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:28.396297 21460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:28.396311 21460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:28.396323 21460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:28.396337 21460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:28.397771 21460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:28.397943 21460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:28.398098 21460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:28.402545 21465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:28.402542 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:28.402563 21466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:28.402838 21460 server_base.cc:1034] running on GCE node
I20260924 03:12:28.403059 21460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:28.403321 21460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:28.404485 21460 hybrid_clock.cc:648] HybridClock initialized: now 1790219548404472 us; error 29 us; skew 500 ppm
Sep 24 03:12:28 dist-test-slave-jtnz krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:28.406782 21460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:28.407122 21460 webserver.cc:460] Webserver started at http://127.0.0.1:39559/ using document root <none> and password file <none>
I20260924 03:12:28.407343 21460 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:28.407404 21460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:28.407491 21460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:28.408362 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "edd4e7ef10064c6c89a4ca6569f54c95"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-jtnz"
server_key: "fb3eb3ca505628f96c535a6a9c2ec1cc"
server_key_iv: "4b61a8725455ed906fd4bc2f07194749"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.408638 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "edd4e7ef10064c6c89a4ca6569f54c95"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-jtnz"
server_key: "fb3eb3ca505628f96c535a6a9c2ec1cc"
server_key_iv: "4b61a8725455ed906fd4bc2f07194749"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.409796 21460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:12:28.410458 21475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:28.410624 21460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:28.410707 21460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "edd4e7ef10064c6c89a4ca6569f54c95"
format_stamp: "Formatted at 2026-09-24 03:12:28 on dist-test-slave-jtnz"
server_key: "fb3eb3ca505628f96c535a6a9c2ec1cc"
server_key_iv: "4b61a8725455ed906fd4bc2f07194749"
server_key_version: "encryptionkey@0"
I20260924 03:12:28.410771 21460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:28.413738 21460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:28.415913 21460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:28.416070 21460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:28.419731 21460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39689
I20260924 03:12:28.419760 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39689 every 8 connection(s)
I20260924 03:12:28.420056 21460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:28.420936 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:28.422986 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95: Bootstrap starting.
I20260924 03:12:28.423512 21528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:28.423779 21528 log.cc:824] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:28.424408 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95: No bootstrap required, opened a new log
I20260924 03:12:28.425880 21528 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260924 03:12:28.426025 21528 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:28.426047 21528 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd4e7ef10064c6c89a4ca6569f54c95, State: Initialized, Role: FOLLOWER
I20260924 03:12:28.426261 21528 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260924 03:12:28.426331 21528 raft_consensus.cc:388] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:28.426378 21528 raft_consensus.cc:482] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:28.426429 21528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:28.426911 21528 raft_consensus.cc:504] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260924 03:12:28.427016 21528 leader_election.cc:302] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd4e7ef10064c6c89a4ca6569f54c95; no voters:
I20260924 03:12:28.427189 21528 leader_election.cc:288] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:28.427212 21531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:28.427347 21531 raft_consensus.cc:686] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [term 1 LEADER]: Becoming Leader. State: Replica: edd4e7ef10064c6c89a4ca6569f54c95, State: Running, Role: LEADER
I20260924 03:12:28.427433 21528 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:28.427469 21531 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } }
I20260924 03:12:28.427616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21460
I20260924 03:12:28.427736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:28.427965 392 external_mini_cluster.cc:1428] Setting key d11499e07a7c02d346797040b604ebe6
I20260924 03:12:28.428179 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd4e7ef10064c6c89a4ca6569f54c95. Latest consensus state: current_term: 1 leader_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } } }
I20260924 03:12:28.428268 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:28.428612 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4e7ef10064c6c89a4ca6569f54c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39689 } } }
I20260924 03:12:28.428681 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:28.428830 21538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:28.429348 21538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:28.430076 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.428416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:56505 (local address 127.0.0.1:39689)
0924 03:12:28.428609 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.428617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.428648 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:12:28.428772 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.428778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.428809 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.429178 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.429591 (+ 413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.429594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.429594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:28.429597 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.429627 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:28.429637 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.429708 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.429901 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.429939 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260924 03:12:28.430758 21538 catalog_manager.cc:1347] Generated new cluster ID: 01753c6532b44a3ca792da1d78be2a7e
I20260924 03:12:28.430809 21538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:28.430963 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:28.435276 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:28.434323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:42527 (local address 127.0.0.1:39689)
0924 03:12:28.434371 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:28.434372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:28.434381 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:28.434400 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:28.434401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:28.434405 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:28.434462 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:28.435036 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:28.435037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:28.435037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:28.435042 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:28.435062 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:28.435066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:28.435112 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:28.435171 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:12:28.435183 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:12:28.438290 21538 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:28.438419 21538 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:28.443310 21538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edd4e7ef10064c6c89a4ca6569f54c95: Generated new TSK 0
I20260924 03:12:28.443449 21538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:28.444754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21460
2026-09-24T03:12:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:28 dist-test-slave-jtnz krb5kdc[21559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:28 dist-test-slave-jtnz krb5kdc[21559](info): set up 2 sockets
Sep 24 03:12:28 dist-test-slave-jtnz krb5kdc[21559](info): commencing operation
krb5kdc: starting...
W20260924 03:12:29.414045 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:29.520525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:29.593701 21572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:29.593868 21572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:29.593895 21572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:29.594938 21572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:29.594976 21572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:29.594990 21572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:29.595024 21572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:29.595052 21572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:29.596365 21572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:29.596566 21572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:29.596748 21572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:29.601094 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:29.601109 21572 server_base.cc:1034] running on GCE node
W20260924 03:12:29.601068 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:29.601080 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:29.601610 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:29.601934 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:29.603080 21572 hybrid_clock.cc:648] HybridClock initialized: now 1790219549603071 us; error 28 us; skew 500 ppm
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:29.605270 21572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:29.605656 21572 webserver.cc:460] Webserver started at http://127.0.0.1:42749/ using document root <none> and password file <none>
I20260924 03:12:29.605859 21572 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:29.605923 21572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:29.606065 21572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:29.607015 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "c9685b2423cf4738bfe76ef63e594918"
format_stamp: "Formatted at 2026-09-24 03:12:29 on dist-test-slave-jtnz"
server_key: "002cb1e6d451bba5ab09d76d46646499"
server_key_iv: "aa0f85af23f35a58cdcf0b729fdfee39"
server_key_version: "encryptionkey@0"
I20260924 03:12:29.607367 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9685b2423cf4738bfe76ef63e594918"
format_stamp: "Formatted at 2026-09-24 03:12:29 on dist-test-slave-jtnz"
server_key: "002cb1e6d451bba5ab09d76d46646499"
server_key_iv: "aa0f85af23f35a58cdcf0b729fdfee39"
server_key_version: "encryptionkey@0"
I20260924 03:12:29.608616 21572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:29.609287 21587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:29.609457 21572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:29.609557 21572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "c9685b2423cf4738bfe76ef63e594918"
format_stamp: "Formatted at 2026-09-24 03:12:29 on dist-test-slave-jtnz"
server_key: "002cb1e6d451bba5ab09d76d46646499"
server_key_iv: "aa0f85af23f35a58cdcf0b729fdfee39"
server_key_version: "encryptionkey@0"
I20260924 03:12:29.609640 21572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:29.623366 21572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:29.624979 21572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:29.625130 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:29.628755 21572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45759
I20260924 03:12:29.628785 21639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45759 every 8 connection(s)
I20260924 03:12:29.629081 21572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:29.629804 21640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:29.631873 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918: Bootstrap starting.
I20260924 03:12:29.632419 21640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:29.632711 21640 log.cc:824] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:29.633332 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918: No bootstrap required, opened a new log
I20260924 03:12:29.634037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21572
I20260924 03:12:29.634142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:29.634346 392 external_mini_cluster.cc:1428] Setting key 2a069bccfe7b918f8123fd476c4e4eb3
I20260924 03:12:29.635375 21640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 03:12:29.635536 21640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:29.635577 21640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9685b2423cf4738bfe76ef63e594918, State: Initialized, Role: FOLLOWER
I20260924 03:12:29.635751 21640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 03:12:29.635839 21640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:29.635881 21640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:29.635934 21640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:29.636818 21640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 03:12:29.636930 21640 leader_election.cc:302] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9685b2423cf4738bfe76ef63e594918; no voters:
I20260924 03:12:29.637132 21640 leader_election.cc:288] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:29.637153 21645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:29.637249 21645 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [term 1 LEADER]: Becoming Leader. State: Replica: c9685b2423cf4738bfe76ef63e594918, State: Running, Role: LEADER
I20260924 03:12:29.637363 21640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:29.637379 21645 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 03:12:29.637763 21646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9685b2423cf4738bfe76ef63e594918. Latest consensus state: current_term: 1 leader_uuid: "c9685b2423cf4738bfe76ef63e594918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260924 03:12:29.637842 21646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:29.638038 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9685b2423cf4738bfe76ef63e594918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9685b2423cf4738bfe76ef63e594918" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260924 03:12:29.638336 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:29.638469 21652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:29.638921 21652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:29.640367 21652 catalog_manager.cc:1347] Generated new cluster ID: 33fd691ddb7943dd9805f8021a90a0a7
I20260924 03:12:29.640424 21652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:12:29.645411 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:29.634822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:42875 (local address 127.0.0.1:45759)
0924 03:12:29.634975 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:29.634981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:29.635006 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:12:29.635361 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:29.635366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:29.635408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:29.635549 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:29.636284 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:29.637251 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:29.638310 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:29.638949 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:29.639071 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:29.641987 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:29.641990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:29.641990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:29.642000 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:29.643814 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:29.644247 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:29.644249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:29.644250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:29.644358 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:29.644614 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:29.644615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:29.644615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:29.644866 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:29.645006 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:29.645156 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:12:29.645242 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 03:12:29.646272 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:29.647876 21652 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:29.648293 21652 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:29.650743 21652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9685b2423cf4738bfe76ef63e594918: Generated new TSK 0
I20260924 03:12:29.650884 21652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:29.653666 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:29.647739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:59819 (local address 127.0.0.1:45759)
0924 03:12:29.647770 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:29.647772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:29.647778 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:12:29.647796 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:29.647796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:29.647804 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:29.647850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:29.648432 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:29.649051 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:29.649949 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:29.650232 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:29.650287 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:29.651490 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:29.651491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:29.651491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:29.651496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:29.652676 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:29.653053 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:29.653053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:29.653053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:29.653097 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:29.653302 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:29.653303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:29.653303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:29.653389 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:29.653447 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:29.653589 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:12:29.653625 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:29.655525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21572
W20260924 03:12:29.659412 21557 connection.cc:535] client connection to 127.0.0.1:45759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45759): Connection reset by peer (error 104)
2026-09-24T03:12:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21671](info): set up 2 sockets
Sep 24 03:12:29 dist-test-slave-jtnz krb5kdc[21671](info): commencing operation
krb5kdc: starting...
W20260924 03:12:30.621987 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:30 dist-test-slave-jtnz krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:30.724702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:30.798460 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:30.798621 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:30.798641 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:30.799589 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:30.799620 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:30.799633 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:30.799644 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:30.799654 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:30.800830 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:30.801017 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:30.801159 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:30.805336 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.805352 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:30.805348 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:30.805492 21684 server_base.cc:1034] running on GCE node
I20260924 03:12:30.805786 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:30.806038 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:30.807178 21684 hybrid_clock.cc:648] HybridClock initialized: now 1790219550807161 us; error 33 us; skew 500 ppm
Sep 24 03:12:30 dist-test-slave-jtnz krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:30.809316 21684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:30.809736 21684 webserver.cc:460] Webserver started at http://127.0.0.1:32997/ using document root <none> and password file <none>
I20260924 03:12:30.809935 21684 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:30.809998 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:30.810137 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:30.811054 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "feaa193bf0484db2a07169a3249ea7e9"
format_stamp: "Formatted at 2026-09-24 03:12:30 on dist-test-slave-jtnz"
server_key: "ac2dabc12c4f749f576fa52c2e8a6f2b"
server_key_iv: "edfe6e9c73ceacd749e403d61d7b9694"
server_key_version: "encryptionkey@0"
I20260924 03:12:30.811375 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "feaa193bf0484db2a07169a3249ea7e9"
format_stamp: "Formatted at 2026-09-24 03:12:30 on dist-test-slave-jtnz"
server_key: "ac2dabc12c4f749f576fa52c2e8a6f2b"
server_key_iv: "edfe6e9c73ceacd749e403d61d7b9694"
server_key_version: "encryptionkey@0"
I20260924 03:12:30.812518 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:30.813210 21699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:30.813374 21684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:30.813482 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "feaa193bf0484db2a07169a3249ea7e9"
format_stamp: "Formatted at 2026-09-24 03:12:30 on dist-test-slave-jtnz"
server_key: "ac2dabc12c4f749f576fa52c2e8a6f2b"
server_key_iv: "edfe6e9c73ceacd749e403d61d7b9694"
server_key_version: "encryptionkey@0"
I20260924 03:12:30.813565 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:30.834530 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:30.836490 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:30.836630 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:30.840241 21684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34769
I20260924 03:12:30.840271 21751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34769 every 8 connection(s)
I20260924 03:12:30.840547 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:30.841379 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:30.843340 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: Bootstrap starting.
I20260924 03:12:30.843847 21752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:30.844100 21752 log.cc:824] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:30.844687 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: No bootstrap required, opened a new log
I20260924 03:12:30.846101 21752 raft_consensus.cc:348] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260924 03:12:30.846239 21752 raft_consensus.cc:374] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:30.846259 21752 raft_consensus.cc:729] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feaa193bf0484db2a07169a3249ea7e9, State: Initialized, Role: FOLLOWER
I20260924 03:12:30.846410 21752 consensus_queue.cc:260] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260924 03:12:30.846464 21752 raft_consensus.cc:388] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:30.846482 21752 raft_consensus.cc:482] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:30.846542 21752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:30.846971 21752 raft_consensus.cc:504] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260924 03:12:30.847049 21752 leader_election.cc:302] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feaa193bf0484db2a07169a3249ea7e9; no voters:
I20260924 03:12:30.847193 21752 leader_election.cc:288] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:30.847257 21755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:30.847368 21755 raft_consensus.cc:686] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [term 1 LEADER]: Becoming Leader. State: Replica: feaa193bf0484db2a07169a3249ea7e9, State: Running, Role: LEADER
I20260924 03:12:30.847414 21752 sys_catalog.cc:564] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:30.847517 21755 consensus_queue.cc:237] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } }
I20260924 03:12:30.848093 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feaa193bf0484db2a07169a3249ea7e9. Latest consensus state: current_term: 1 leader_uuid: "feaa193bf0484db2a07169a3249ea7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260924 03:12:30.848182 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:30.848364 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feaa193bf0484db2a07169a3249ea7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feaa193bf0484db2a07169a3249ea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34769 } } }
I20260924 03:12:30.848448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21684
I20260924 03:12:30.848492 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:30.848556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:30.848755 392 external_mini_cluster.cc:1428] Setting key 860781eb06655eb57d458f0604a04501
W20260924 03:12:30.849238 21771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:30.849372 21771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:30.849449 21764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:30.849607 21764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:30.851038 21764 catalog_manager.cc:1347] Generated new cluster ID: 34649e4e239d41af96502e54cbcc16b4
I20260924 03:12:30.851084 21764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:30.854385 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.849160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:54865 (local address 127.0.0.1:34769)
0924 03:12:30.849350 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:30.849357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:30.849543 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:12:30.849664 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:30.849670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:30.849723 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:30.849856 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:30.850068 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.851352 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.852390 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.852984 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.853088 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.853698 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:30.853702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:30.853702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:30.853871 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:30.853901 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:30.853913 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:30.853997 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:30.854066 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 03:12:30.854194 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260924 03:12:30.855157 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:30.861734 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:30.858495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:37243 (local address 127.0.0.1:34769)
0924 03:12:30.858531 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:30.858533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:30.858766 (+ 233us) server_negotiation.cc:407] Connection header received
0924 03:12:30.858793 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:30.858794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:30.858803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:30.858864 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:30.858964 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.859657 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.860519 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:30.860839 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:30.860924 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:30.861509 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:30.861510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:30.861510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:30.861514 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:30.861527 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:30.861531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:30.861590 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:30.861655 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:12:30.861666 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:30.883599 21764 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:30.884258 21764 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:30.890394 21764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feaa193bf0484db2a07169a3249ea7e9: Generated new TSK 0
I20260924 03:12:30.890548 21764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:30.896282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21684
W20260924 03:12:30.900461 21669 connection.cc:535] client connection to 127.0.0.1:34769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34769): Connection reset by peer (error 104)
2026-09-24T03:12:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:30 dist-test-slave-jtnz krb5kdc[21783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:30 dist-test-slave-jtnz krb5kdc[21783](info): set up 2 sockets
Sep 24 03:12:30 dist-test-slave-jtnz krb5kdc[21783](info): commencing operation
krb5kdc: starting...
W20260924 03:12:31.902164 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:31 dist-test-slave-jtnz krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:12:32.004555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:32.077467 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:32.077630 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:32.077651 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:32.078830 21796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:32.078869 21796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:32.078882 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:32.078894 21796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:32.078938 21796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:32.080228 21796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:32.080457 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:32.080657 21796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:32.085045 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:32.085050 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:32.085114 21796 server_base.cc:1034] running on GCE node
W20260924 03:12:32.085054 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:32.085450 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:32.085723 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:32.086874 21796 hybrid_clock.cc:648] HybridClock initialized: now 1790219552086853 us; error 33 us; skew 500 ppm
Sep 24 03:12:32 dist-test-slave-jtnz krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:32.088883 21796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:12:32.089250 21796 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260924 03:12:32.089443 21796 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:32.089501 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:32.089638 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:32.090579 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "df9b71a7595a42a39fedcb7ee23c02c6"
format_stamp: "Formatted at 2026-09-24 03:12:32 on dist-test-slave-jtnz"
server_key: "9775774fd0044d732667407789398e34"
server_key_iv: "c60884c8001b6f4adefdeb7707aa1ed0"
server_key_version: "encryptionkey@0"
I20260924 03:12:32.090873 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "df9b71a7595a42a39fedcb7ee23c02c6"
format_stamp: "Formatted at 2026-09-24 03:12:32 on dist-test-slave-jtnz"
server_key: "9775774fd0044d732667407789398e34"
server_key_iv: "c60884c8001b6f4adefdeb7707aa1ed0"
server_key_version: "encryptionkey@0"
I20260924 03:12:32.091931 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:32.092545 21811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:32.092700 21796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:32.092790 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "df9b71a7595a42a39fedcb7ee23c02c6"
format_stamp: "Formatted at 2026-09-24 03:12:32 on dist-test-slave-jtnz"
server_key: "9775774fd0044d732667407789398e34"
server_key_iv: "c60884c8001b6f4adefdeb7707aa1ed0"
server_key_version: "encryptionkey@0"
I20260924 03:12:32.092864 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:32.095685 21796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:32.097441 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:32.097581 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:32.101228 21796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37309
I20260924 03:12:32.101263 21863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37309 every 8 connection(s)
I20260924 03:12:32.101538 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:32.102475 21864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:32.104378 21864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: Bootstrap starting.
I20260924 03:12:32.104844 21864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:32.105070 21864 log.cc:824] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:32.105652 21864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: No bootstrap required, opened a new log
I20260924 03:12:32.107074 21864 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 03:12:32.107187 21864 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:32.107205 21864 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9b71a7595a42a39fedcb7ee23c02c6, State: Initialized, Role: FOLLOWER
I20260924 03:12:32.107316 21864 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 03:12:32.107383 21864 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:32.107403 21864 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:32.107427 21864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:32.107831 21864 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 03:12:32.107900 21864 leader_election.cc:302] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9b71a7595a42a39fedcb7ee23c02c6; no voters:
I20260924 03:12:32.108008 21864 leader_election.cc:288] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:32.108078 21867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:32.108210 21867 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [term 1 LEADER]: Becoming Leader. State: Replica: df9b71a7595a42a39fedcb7ee23c02c6, State: Running, Role: LEADER
I20260924 03:12:32.108286 21867 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 03:12:32.108394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21796
I20260924 03:12:32.108506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:32.108717 392 external_mini_cluster.cc:1428] Setting key bd5f5d65fa2e67590c4d6a5da313a41e
I20260924 03:12:32.108677 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9b71a7595a42a39fedcb7ee23c02c6. Latest consensus state: current_term: 1 leader_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260924 03:12:32.108757 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:32.108881 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9b71a7595a42a39fedcb7ee23c02c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260924 03:12:32.108938 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:32.109093 21864 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:12:32.110677 21885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:32.110771 21885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:32.110862 21871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:32.111002 21871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:32.111656 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.109107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:49221 (local address 127.0.0.1:37309)
0924 03:12:32.109275 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:32.109284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:32.110381 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 03:12:32.110502 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:32.110508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:32.110535 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:32.110654 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:32.111207 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:32.111210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:32.111211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:32.111214 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:32.111247 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:32.111259 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:32.111339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:32.111406 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:12:32.111448 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 03:12:32.112516 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:32.113824 21871 catalog_manager.cc:1347] Generated new cluster ID: dd857a1e716f46408fb9b113c960870f
I20260924 03:12:32.113875 21871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:32.115113 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:32.113916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.76:57221 (local address 127.0.0.1:37309)
0924 03:12:32.113949 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:32.113951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:32.114161 (+ 210us) server_negotiation.cc:407] Connection header received
0924 03:12:32.114246 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:32.114247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:32.114252 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:32.114303 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:32.114902 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:32.114903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:32.114903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:32.114907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:32.114919 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:32.114922 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:32.114986 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:32.115040 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:12:32.115054 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:32.131695 21871 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:32.131829 21871 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:32.135344 21871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9b71a7595a42a39fedcb7ee23c02c6: Generated new TSK 0
I20260924 03:12:32.135497 21871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:32.140317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21796
2026-09-24T03:12:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1243 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:32 dist-test-slave-jtnz krb5kdc[21895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:32 dist-test-slave-jtnz krb5kdc[21895](info): set up 2 sockets
Sep 24 03:12:32 dist-test-slave-jtnz krb5kdc[21895](info): commencing operation
krb5kdc: starting...
W20260924 03:12:33.121984 392 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:33.224390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:33.298415 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:33.298614 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:33.298636 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:33.299783 21908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:33.299826 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:33.299840 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:33.299853 21908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:33.299891 21908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:33.301345 21908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:33.301592 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:33.301808 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:33.306403 21913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:33.306439 21914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:33.306481 21908 server_base.cc:1034] running on GCE node
W20260924 03:12:33.306439 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:33.306759 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:33.306988 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:33.308121 21908 hybrid_clock.cc:648] HybridClock initialized: now 1790219553308103 us; error 40 us; skew 500 ppm
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:33.310431 21908 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:33.310865 21908 webserver.cc:460] Webserver started at http://127.0.98.62:34209/ using document root <none> and password file <none>
I20260924 03:12:33.311115 21908 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:33.311158 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:33.311308 21908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:33.312281 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "2992c843af99442daa49a4ab76e83077"
format_stamp: "Formatted at 2026-09-24 03:12:33 on dist-test-slave-jtnz"
server_key: "ebdec882b04a07ac68248ce99520d4d7"
server_key_iv: "4108579aafea5e4d47c3b60401c3a2cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:33.312636 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "2992c843af99442daa49a4ab76e83077"
format_stamp: "Formatted at 2026-09-24 03:12:33 on dist-test-slave-jtnz"
server_key: "ebdec882b04a07ac68248ce99520d4d7"
server_key_iv: "4108579aafea5e4d47c3b60401c3a2cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:33.313877 21908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:33.314615 21923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.314790 21908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:33.314859 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "2992c843af99442daa49a4ab76e83077"
format_stamp: "Formatted at 2026-09-24 03:12:33 on dist-test-slave-jtnz"
server_key: "ebdec882b04a07ac68248ce99520d4d7"
server_key_iv: "4108579aafea5e4d47c3b60401c3a2cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:33.314937 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:33.323779 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:33.325317 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:33.325454 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:33.329072 21908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40539
I20260924 03:12:33.329138 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40539 every 8 connection(s)
I20260924 03:12:33.329448 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:33.330300 21976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:33.332332 21976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077: Bootstrap starting.
I20260924 03:12:33.332886 21976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:33.333185 21976 log.cc:824] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:33.333822 21976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077: No bootstrap required, opened a new log
I20260924 03:12:33.335335 21976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260924 03:12:33.335477 21976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:33.335521 21976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2992c843af99442daa49a4ab76e83077, State: Initialized, Role: FOLLOWER
I20260924 03:12:33.335695 21976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260924 03:12:33.335790 21976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:33.335832 21976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:33.335922 21976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:33.336395 21976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260924 03:12:33.336508 21976 leader_election.cc:302] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2992c843af99442daa49a4ab76e83077; no voters:
I20260924 03:12:33.336673 21976 leader_election.cc:288] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:33.336692 21979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:33.336849 21979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [term 1 LEADER]: Becoming Leader. State: Replica: 2992c843af99442daa49a4ab76e83077, State: Running, Role: LEADER
I20260924 03:12:33.336905 21976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:33.336977 21979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260924 03:12:33.337332 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2992c843af99442daa49a4ab76e83077. Latest consensus state: current_term: 1 leader_uuid: "2992c843af99442daa49a4ab76e83077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } } }
I20260924 03:12:33.337419 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:33.337441 21980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2992c843af99442daa49a4ab76e83077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2992c843af99442daa49a4ab76e83077" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } } }
I20260924 03:12:33.337496 21980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:33.337630 21988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:33.338145 21988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:33.338395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 21908
I20260924 03:12:33.338505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:33.338694 392 external_mini_cluster.cc:1428] Setting key c1f4e2a89a602d86420ea6c3bf0afefd
I20260924 03:12:33.340188 21988 catalog_manager.cc:1347] Generated new cluster ID: 15c3eade1f214164abbc797095757dc7
I20260924 03:12:33.340240 21988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[21895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:33.352641 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.339336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56773 (local address 127.0.98.62:40539)
0924 03:12:33.341842 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:33.341850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:33.342089 (+ 239us) server_negotiation.cc:407] Connection header received
0924 03:12:33.342212 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:33.342219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:33.342268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:33.342371 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:33.343166 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:33.344486 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:33.345540 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:33.346139 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:33.346258 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:33.349069 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:33.349073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:33.349074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:33.349086 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:33.351025 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:33.351426 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:33.351429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:33.351430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:33.351547 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:33.351776 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:33.351777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:33.351777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:33.352059 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:33.352202 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:33.352303 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:12:33.352400 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2467,"thread_start_us":2395,"threads_started":1}
I20260924 03:12:33.353504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:33.361368 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:33.354901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44875 (local address 127.0.98.62:40539)
0924 03:12:33.354932 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:33.354934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:33.354992 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:12:33.355042 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:33.355042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:33.355050 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:33.355113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:33.355618 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:33.356317 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:33.357221 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:33.357534 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:33.357600 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:33.358804 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:33.358805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:33.358805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:33.358812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:33.360240 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:33.360607 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:33.360608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:33.360608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:33.360663 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:33.360878 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:33.360879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:33.360879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:33.361014 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:33.361122 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:33.361226 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:12:33.361283 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:33.381036 21988 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:33.381699 21988 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:33.386384 21988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2992c843af99442daa49a4ab76e83077: Generated new TSK 0
I20260924 03:12:33.386545 21988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:33.394631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 21908
W20260924 03:12:33.399263 21893 connection.cc:535] client connection to 127.0.98.62:40539 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40539): Connection reset by peer (error 104)
2026-09-24T03:12:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[22007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[22007](info): set up 2 sockets
Sep 24 03:12:33 dist-test-slave-jtnz krb5kdc[22007](info): commencing operation
krb5kdc: starting...
W20260924 03:12:34.366277 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:34 dist-test-slave-jtnz krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:34.473070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:34.552944 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:34.553154 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:34.553194 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:34.554383 22020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:34.554440 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:34.554464 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:34.554495 22020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:34.554523 22020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:34.555923 22020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:34.556173 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:34.556428 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:34.560937 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:34.560945 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:34.560974 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:34.561271 22020 server_base.cc:1034] running on GCE node
I20260924 03:12:34.561539 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:34.561803 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:34.562935 22020 hybrid_clock.cc:648] HybridClock initialized: now 1790219554562914 us; error 36 us; skew 500 ppm
Sep 24 03:12:34 dist-test-slave-jtnz krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:34.565320 22020 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:34.565759 22020 webserver.cc:460] Webserver started at http://127.0.98.62:33753/ using document root <none> and password file <none>
I20260924 03:12:34.565987 22020 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:34.566054 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:34.566222 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:34.567260 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "8cd220b386f942ef814075b6a22276a1"
format_stamp: "Formatted at 2026-09-24 03:12:34 on dist-test-slave-jtnz"
server_key: "59a31497b1edfd87b80ec1d60af8bdb8"
server_key_iv: "a9e3eee5472dbc87567499cf61342905"
server_key_version: "encryptionkey@0"
I20260924 03:12:34.567596 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cd220b386f942ef814075b6a22276a1"
format_stamp: "Formatted at 2026-09-24 03:12:34 on dist-test-slave-jtnz"
server_key: "59a31497b1edfd87b80ec1d60af8bdb8"
server_key_iv: "a9e3eee5472dbc87567499cf61342905"
server_key_version: "encryptionkey@0"
I20260924 03:12:34.568877 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:34.569556 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:34.569731 22020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:34.569825 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "8cd220b386f942ef814075b6a22276a1"
format_stamp: "Formatted at 2026-09-24 03:12:34 on dist-test-slave-jtnz"
server_key: "59a31497b1edfd87b80ec1d60af8bdb8"
server_key_iv: "a9e3eee5472dbc87567499cf61342905"
server_key_version: "encryptionkey@0"
I20260924 03:12:34.569909 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:34.591501 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:34.593675 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:34.593830 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:34.598119 22020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35723
I20260924 03:12:34.598168 22087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35723 every 8 connection(s)
I20260924 03:12:34.598511 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:34.599283 22088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:34.601318 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1: Bootstrap starting.
I20260924 03:12:34.601815 22088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:34.602043 22088 log.cc:824] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:34.602646 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1: No bootstrap required, opened a new log
I20260924 03:12:34.604086 22088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } }
I20260924 03:12:34.604205 22088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:34.604226 22088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd220b386f942ef814075b6a22276a1, State: Initialized, Role: FOLLOWER
I20260924 03:12:34.604341 22088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } }
I20260924 03:12:34.604389 22088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:34.604408 22088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:34.604487 22088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:34.604966 22088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } }
I20260924 03:12:34.605075 22088 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd220b386f942ef814075b6a22276a1; no voters:
I20260924 03:12:34.605247 22088 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:34.605261 22091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:34.605432 22091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd220b386f942ef814075b6a22276a1, State: Running, Role: LEADER
I20260924 03:12:34.605492 22088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:34.605582 22091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } }
I20260924 03:12:34.605975 22093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd220b386f942ef814075b6a22276a1. Latest consensus state: current_term: 1 leader_uuid: "8cd220b386f942ef814075b6a22276a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } } }
I20260924 03:12:34.606086 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd220b386f942ef814075b6a22276a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd220b386f942ef814075b6a22276a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35723 } } }
I20260924 03:12:34.606112 22093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:34.606195 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:34.606644 22100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:34.607121 22100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:34.607116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22020
I20260924 03:12:34.607224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:34.607431 392 external_mini_cluster.cc:1428] Setting key 73893ebd9bc7d7ad9224ebfc20d29792
I20260924 03:12:34.608786 22100 catalog_manager.cc:1347] Generated new cluster ID: 9cfe57c6408f412cb44cf8214b832d46
I20260924 03:12:34.608837 22100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:34.612634 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.607811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46255 (local address 127.0.98.62:35723)
0924 03:12:34.607968 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:34.607986 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:34.608056 (+ 70us) server_negotiation.cc:407] Connection header received
0924 03:12:34.608205 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:34.608211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:34.608415 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:34.608886 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:34.608982 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.609856 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.610742 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.611158 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.611298 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.612007 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:34.612011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:34.612011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:34.612191 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:34.612219 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:34.612230 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:34.612322 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:34.612407 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:12:34.612436 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260924 03:12:34.613647 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:34.617592 22100 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:34.617798 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:34.614998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58651 (local address 127.0.98.62:35723)
0924 03:12:34.615031 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:34.615033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:34.615165 (+ 132us) server_negotiation.cc:407] Connection header received
0924 03:12:34.615229 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:34.615230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:34.615240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:34.615281 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:34.615411 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.615864 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.616548 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:34.616779 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:34.616860 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:34.617477 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:34.617479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:34.617479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:34.617483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:34.617496 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:34.617500 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:34.617547 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:34.617728 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:12:34.617738 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:34.618268 22100 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:34.624881 22100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd220b386f942ef814075b6a22276a1: Generated new TSK 0
I20260924 03:12:34.625034 22100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:34.628022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22020
W20260924 03:12:34.632175 22005 connection.cc:535] client connection to 127.0.98.62:35723 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35723): Connection reset by peer (error 104)
2026-09-24T03:12:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:34 dist-test-slave-jtnz krb5kdc[22119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:34 dist-test-slave-jtnz krb5kdc[22119](info): set up 2 sockets
Sep 24 03:12:34 dist-test-slave-jtnz krb5kdc[22119](info): commencing operation
krb5kdc: starting...
W20260924 03:12:35.602083 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:35 dist-test-slave-jtnz krb5kdc[22119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:35.707054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:35.778141 22132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:35.778347 22132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:35.778367 22132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:35.779294 22132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:35.779323 22132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:35.779335 22132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:35.779346 22132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:35.779356 22132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:35.780511 22132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:35.780663 22132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:35.780788 22132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:35.785202 22137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:35.785239 22138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:35.785214 22140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:35.785468 22132 server_base.cc:1034] running on GCE node
I20260924 03:12:35.785657 22132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:35.785859 22132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:35.786989 22132 hybrid_clock.cc:648] HybridClock initialized: now 1790219555786976 us; error 26 us; skew 500 ppm
Sep 24 03:12:35 dist-test-slave-jtnz krb5kdc[22119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:35.789187 22132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:35.789528 22132 webserver.cc:460] Webserver started at http://127.0.98.62:32995/ using document root <none> and password file <none>
I20260924 03:12:35.789707 22132 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:35.789743 22132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:35.789860 22132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:35.790814 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "1f4924ee2893418a9b3e971800f93de9"
format_stamp: "Formatted at 2026-09-24 03:12:35 on dist-test-slave-jtnz"
server_key: "0b9014948d4972c5b5a08fc4c603d7c0"
server_key_iv: "eddccbbd7d9243fd41654be29da5ac17"
server_key_version: "encryptionkey@0"
I20260924 03:12:35.791123 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f4924ee2893418a9b3e971800f93de9"
format_stamp: "Formatted at 2026-09-24 03:12:35 on dist-test-slave-jtnz"
server_key: "0b9014948d4972c5b5a08fc4c603d7c0"
server_key_iv: "eddccbbd7d9243fd41654be29da5ac17"
server_key_version: "encryptionkey@0"
I20260924 03:12:35.792212 22132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:35.792829 22147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:35.792999 22132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:35.793094 22132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "1f4924ee2893418a9b3e971800f93de9"
format_stamp: "Formatted at 2026-09-24 03:12:35 on dist-test-slave-jtnz"
server_key: "0b9014948d4972c5b5a08fc4c603d7c0"
server_key_iv: "eddccbbd7d9243fd41654be29da5ac17"
server_key_version: "encryptionkey@0"
I20260924 03:12:35.793165 22132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:35.796089 22132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:35.797603 22132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:35.797730 22132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:35.801220 22132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40739
I20260924 03:12:35.801247 22199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40739 every 8 connection(s)
I20260924 03:12:35.801532 22132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:35.802517 22200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:35.804384 22200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9: Bootstrap starting.
I20260924 03:12:35.804807 22200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:35.805001 22200 log.cc:824] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:35.805518 22200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9: No bootstrap required, opened a new log
I20260924 03:12:35.806949 22200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } }
I20260924 03:12:35.807061 22200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:35.807081 22200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4924ee2893418a9b3e971800f93de9, State: Initialized, Role: FOLLOWER
I20260924 03:12:35.807192 22200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } }
I20260924 03:12:35.807289 22200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:35.807309 22200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:35.807333 22200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:35.807735 22200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } }
I20260924 03:12:35.807803 22200 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4924ee2893418a9b3e971800f93de9; no voters:
I20260924 03:12:35.807899 22200 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:35.807955 22203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:35.808082 22203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4924ee2893418a9b3e971800f93de9, State: Running, Role: LEADER
I20260924 03:12:35.808125 22200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:35.808153 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } }
I20260924 03:12:35.808473 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4924ee2893418a9b3e971800f93de9. Latest consensus state: current_term: 1 leader_uuid: "1f4924ee2893418a9b3e971800f93de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } } }
I20260924 03:12:35.808550 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:35.808584 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4924ee2893418a9b3e971800f93de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4924ee2893418a9b3e971800f93de9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40739 } } }
I20260924 03:12:35.808640 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:35.809173 22209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:35.809544 22209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:35.810796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22132
I20260924 03:12:35.810899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:35.811030 22209 catalog_manager.cc:1347] Generated new cluster ID: f7ec26c60e0041bfa35db8fa7776af5a
I20260924 03:12:35.811059 392 external_mini_cluster.cc:1428] Setting key 21ba3ebea76358ef9f8aa5eeec29fdea
I20260924 03:12:35.811079 22209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:35.815620 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:35.811407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41987 (local address 127.0.98.62:40739)
0924 03:12:35.811549 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:35.811557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:35.814383 (+ 2826us) server_negotiation.cc:407] Connection header received
0924 03:12:35.814516 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:35.814521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:35.814547 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:35.814667 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:35.815225 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:35.815228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:35.815228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:35.815231 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:35.815263 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:35.815280 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:35.815340 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:35.815402 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 03:12:35.815437 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 03:12:35.816529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:35.819092 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:35.817770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57447 (local address 127.0.98.62:40739)
0924 03:12:35.817802 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:35.817803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:35.818303 (+ 500us) server_negotiation.cc:407] Connection header received
0924 03:12:35.818367 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:35.818368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:35.818373 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:35.818422 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:35.818870 (+ 448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:35.818871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:35.818871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:35.818875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:35.818885 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:35.818889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:35.818938 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:35.818994 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:12:35.819005 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:35.823657 22209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:35.823788 22209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:35.829424 22209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4924ee2893418a9b3e971800f93de9: Generated new TSK 0
I20260924 03:12:35.829558 22209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:35.832489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22132
2026-09-24T03:12:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:35 dist-test-slave-jtnz krb5kdc[22231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:35 dist-test-slave-jtnz krb5kdc[22231](info): set up 2 sockets
Sep 24 03:12:35 dist-test-slave-jtnz krb5kdc[22231](info): commencing operation
krb5kdc: starting...
W20260924 03:12:36.773870 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:36 dist-test-slave-jtnz krb5kdc[22231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:36.876673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:36.948359 22244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:36.948555 22244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:36.948590 22244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:36.949661 22244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:36.949718 22244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:36.949740 22244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:36.949766 22244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:36.949790 22244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:36.951064 22244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:36.951293 22244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:36.951496 22244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:36.955714 22249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:36.955741 22252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:36.955719 22250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:36.955905 22244 server_base.cc:1034] running on GCE node
I20260924 03:12:36.956103 22244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:36.956372 22244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:36.957526 22244 hybrid_clock.cc:648] HybridClock initialized: now 1790219556957512 us; error 29 us; skew 500 ppm
Sep 24 03:12:36 dist-test-slave-jtnz krb5kdc[22231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:36.967790 22244 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:36.968133 22244 webserver.cc:460] Webserver started at http://127.0.98.62:40327/ using document root <none> and password file <none>
I20260924 03:12:36.968314 22244 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:36.968387 22244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:36.968544 22244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:36.969424 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3"
format_stamp: "Formatted at 2026-09-24 03:12:36 on dist-test-slave-jtnz"
server_key: "a38c66012a558d233d954b43f96cd2e4"
server_key_iv: "4159437aae5d92976e4b43ab65242844"
server_key_version: "encryptionkey@0"
I20260924 03:12:36.969741 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3"
format_stamp: "Formatted at 2026-09-24 03:12:36 on dist-test-slave-jtnz"
server_key: "a38c66012a558d233d954b43f96cd2e4"
server_key_iv: "4159437aae5d92976e4b43ab65242844"
server_key_version: "encryptionkey@0"
I20260924 03:12:36.970954 22244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:36.971518 22259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:36.971657 22244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:36.971740 22244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3"
format_stamp: "Formatted at 2026-09-24 03:12:36 on dist-test-slave-jtnz"
server_key: "a38c66012a558d233d954b43f96cd2e4"
server_key_iv: "4159437aae5d92976e4b43ab65242844"
server_key_version: "encryptionkey@0"
I20260924 03:12:36.971807 22244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:36.991783 22244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:36.993798 22244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:36.993947 22244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:36.997254 22244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40487
I20260924 03:12:36.997279 22311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40487 every 8 connection(s)
I20260924 03:12:36.997553 22244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:36.998322 22312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:37.000135 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3: Bootstrap starting.
I20260924 03:12:37.000336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22244
I20260924 03:12:37.000442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:37.000592 392 external_mini_cluster.cc:1428] Setting key 89a64c2b007fa70917bf6169d346f8ce
I20260924 03:12:37.000727 22312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:37.001080 22312 log.cc:824] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:37.001803 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3: No bootstrap required, opened a new log
I20260924 03:12:37.004076 22312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } }
I20260924 03:12:37.004223 22312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:37.004264 22312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1373a0dd39bb4a5ca14d3701bfa89ee3, State: Initialized, Role: FOLLOWER
I20260924 03:12:37.004384 22312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } }
I20260924 03:12:37.004482 22312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:37.004522 22312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:37.004568 22312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:37.005074 22312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } }
I20260924 03:12:37.005169 22312 leader_election.cc:302] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1373a0dd39bb4a5ca14d3701bfa89ee3; no voters:
I20260924 03:12:37.005309 22312 leader_election.cc:288] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:37.005365 22317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:37.005505 22312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:37.005527 22317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 1373a0dd39bb4a5ca14d3701bfa89ee3, State: Running, Role: LEADER
I20260924 03:12:37.005611 22317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } }
I20260924 03:12:37.005918 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1373a0dd39bb4a5ca14d3701bfa89ee3. Latest consensus state: current_term: 1 leader_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } } }
I20260924 03:12:37.006060 22318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1373a0dd39bb4a5ca14d3701bfa89ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40487 } } }
I20260924 03:12:37.006242 22318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:37.006131 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:37.007503 22326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 03:12:37 dist-test-slave-jtnz krb5kdc[22231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:37.008205 22326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:37.009701 22326 catalog_manager.cc:1347] Generated new cluster ID: ec216eda9c6847d3a2aa4535a4f012a0
I20260924 03:12:37.009752 22326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:37.011803 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:37.001071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55033 (local address 127.0.98.62:40487)
0924 03:12:37.001316 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:37.001325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:37.001351 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:12:37.001957 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:37.001965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:37.002021 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:37.002156 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:37.002773 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:37.004189 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:37.005033 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:37.005624 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:37.006037 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:37.008390 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:37.008393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:37.008393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:37.008404 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:37.010383 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:37.010726 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:37.010729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:37.010729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:37.010810 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:37.011013 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:37.011013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:37.011014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:37.011220 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:37.011394 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:37.011487 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:12:37.011609 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260924 03:12:37.012567 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:37.019526 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:37.014038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57807 (local address 127.0.98.62:40487)
0924 03:12:37.014088 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:37.014090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:37.014306 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:12:37.014370 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:37.014371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:37.014380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:37.014435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:37.014867 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:37.015292 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:37.015915 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:37.016110 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:37.016187 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:37.017212 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:37.017213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:37.017213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:37.017219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:37.018501 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:37.018829 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:37.018830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:37.018830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:37.018877 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:37.019074 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:37.019074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:37.019075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:37.019171 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:37.019258 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:37.019360 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 03:12:37.019424 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:12:37.024433 22326 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:37.024923 22326 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:37.030102 22326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1373a0dd39bb4a5ca14d3701bfa89ee3: Generated new TSK 0
I20260924 03:12:37.030295 22326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:37.033224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22244
2026-09-24T03:12:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:37 dist-test-slave-jtnz krb5kdc[22343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:37 dist-test-slave-jtnz krb5kdc[22343](info): set up 2 sockets
Sep 24 03:12:37 dist-test-slave-jtnz krb5kdc[22343](info): commencing operation
krb5kdc: starting...
W20260924 03:12:37.981884 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:38 dist-test-slave-jtnz krb5kdc[22343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:38.084719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:38.156239 22356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:38.156391 22356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:38.156410 22356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:38.157361 22356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:38.157397 22356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:38.157408 22356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:38.157419 22356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:38.157465 22356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:38.158778 22356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:38.158989 22356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:38.159188 22356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:38.163422 22361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:38.163565 22356 server_base.cc:1034] running on GCE node
W20260924 03:12:38.163414 22364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:38.163412 22362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:38.163893 22356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:38.164113 22356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:38.165277 22356 hybrid_clock.cc:648] HybridClock initialized: now 1790219558165277 us; error 35 us; skew 500 ppm
Sep 24 03:12:38 dist-test-slave-jtnz krb5kdc[22343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:38.167388 22356 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:38.167783 22356 webserver.cc:460] Webserver started at http://127.0.98.62:46131/ using document root <none> and password file <none>
I20260924 03:12:38.167982 22356 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:38.168043 22356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:38.168176 22356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:38.169075 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "60e22b2ed4674afdb2d07a99f361603e"
format_stamp: "Formatted at 2026-09-24 03:12:38 on dist-test-slave-jtnz"
server_key: "662b151cfe0875aaffa8f05a18957761"
server_key_iv: "5f4abc9a92d04f2570988af7c180b921"
server_key_version: "encryptionkey@0"
I20260924 03:12:38.169382 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "60e22b2ed4674afdb2d07a99f361603e"
format_stamp: "Formatted at 2026-09-24 03:12:38 on dist-test-slave-jtnz"
server_key: "662b151cfe0875aaffa8f05a18957761"
server_key_iv: "5f4abc9a92d04f2570988af7c180b921"
server_key_version: "encryptionkey@0"
I20260924 03:12:38.170583 22356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:12:38.171204 22371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:38.171360 22356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:38.171449 22356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "60e22b2ed4674afdb2d07a99f361603e"
format_stamp: "Formatted at 2026-09-24 03:12:38 on dist-test-slave-jtnz"
server_key: "662b151cfe0875aaffa8f05a18957761"
server_key_iv: "5f4abc9a92d04f2570988af7c180b921"
server_key_version: "encryptionkey@0"
I20260924 03:12:38.171526 22356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:38.182646 22356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:38.184195 22356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:38.184342 22356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:38.187932 22356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35763
I20260924 03:12:38.187979 22423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35763 every 8 connection(s)
I20260924 03:12:38.188277 22356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:38.188990 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:38.190944 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e: Bootstrap starting.
I20260924 03:12:38.191409 22424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:38.191633 22424 log.cc:824] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:38.192220 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e: No bootstrap required, opened a new log
I20260924 03:12:38.193585 22424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } }
I20260924 03:12:38.193702 22424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:38.193722 22424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e22b2ed4674afdb2d07a99f361603e, State: Initialized, Role: FOLLOWER
I20260924 03:12:38.193809 22424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } }
I20260924 03:12:38.193852 22424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:38.193871 22424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:38.193902 22424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:38.194403 22424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } }
I20260924 03:12:38.194484 22424 leader_election.cc:302] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e22b2ed4674afdb2d07a99f361603e; no voters:
I20260924 03:12:38.194586 22424 leader_election.cc:288] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:38.194645 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:38.194779 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [term 1 LEADER]: Becoming Leader. State: Replica: 60e22b2ed4674afdb2d07a99f361603e, State: Running, Role: LEADER
I20260924 03:12:38.194819 22424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:38.194881 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } }
I20260924 03:12:38.195243 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e22b2ed4674afdb2d07a99f361603e. Latest consensus state: current_term: 1 leader_uuid: "60e22b2ed4674afdb2d07a99f361603e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } }
I20260924 03:12:38.195312 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:38.195331 22428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e22b2ed4674afdb2d07a99f361603e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e22b2ed4674afdb2d07a99f361603e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } }
I20260924 03:12:38.195408 22428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:38.195672 22435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:38.196099 22435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:38.197325 22435 catalog_manager.cc:1347] Generated new cluster ID: 3f11996a594c406ab65b90013d9c8854
I20260924 03:12:38.197373 22435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:38.198446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22356
I20260924 03:12:38.198554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:38.198709 392 external_mini_cluster.cc:1428] Setting key 4c013f36d4225f80d582da7032bf5d4b
I20260924 03:12:38.204432 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:38.199177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60121 (local address 127.0.98.62:35763)
0924 03:12:38.199322 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:38.199346 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:38.199370 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:12:38.199476 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:38.199481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:38.199535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:38.199648 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:38.199851 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:38.201122 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:38.202351 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:38.202938 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:38.203032 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:38.203692 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:38.203695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:38.203696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:38.203871 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:38.203907 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:38.203920 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:38.203989 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:38.204080 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:12:38.204217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260924 03:12:38.204728 22435 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:38.205230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:38.205283 22435 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:38.207692 22435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e22b2ed4674afdb2d07a99f361603e: Generated new TSK 0
I20260924 03:12:38.207835 22435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:38.212913 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:38.206564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44637 (local address 127.0.98.62:35763)
0924 03:12:38.206667 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:38.206669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:38.210333 (+ 3664us) server_negotiation.cc:407] Connection header received
0924 03:12:38.210373 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:38.210374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:38.210383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:38.210447 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:38.210517 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:38.211221 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:38.211832 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:38.212104 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:38.212163 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:38.212693 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:38.212695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:38.212695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:38.212699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:38.212721 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:38.212725 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:38.212782 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:38.212813 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 03:12:38.212833 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260924 03:12:38.214571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22356
2026-09-24T03:12:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:38 dist-test-slave-jtnz krb5kdc[22456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:38 dist-test-slave-jtnz krb5kdc[22456](info): set up 2 sockets
Sep 24 03:12:38 dist-test-slave-jtnz krb5kdc[22456](info): commencing operation
krb5kdc: starting...
W20260924 03:12:39.173743 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:39 dist-test-slave-jtnz krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:39.274267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:39.351521 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:39.351727 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:39.351764 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:39.352856 22469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:39.352919 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:39.352945 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:39.352975 22469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:39.352998 22469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:39.354498 22469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:39.354730 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:39.354945 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:39.359476 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:39.359454 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:39.359470 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:39.359571 22469 server_base.cc:1034] running on GCE node
I20260924 03:12:39.359825 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:39.360067 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:39.361208 22469 hybrid_clock.cc:648] HybridClock initialized: now 1790219559361198 us; error 27 us; skew 500 ppm
Sep 24 03:12:39 dist-test-slave-jtnz krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:39.363636 22469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:39.363986 22469 webserver.cc:460] Webserver started at http://127.0.98.62:36911/ using document root <none> and password file <none>
I20260924 03:12:39.364146 22469 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:39.364184 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:39.364264 22469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:39.365141 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "5f35d6f5a16c4c3f9d01639750674d39"
format_stamp: "Formatted at 2026-09-24 03:12:39 on dist-test-slave-jtnz"
server_key: "79409122309c714283a71632d85eb323"
server_key_iv: "2248e6e15197d957f3bca67ad5b0060e"
server_key_version: "encryptionkey@0"
I20260924 03:12:39.365406 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f35d6f5a16c4c3f9d01639750674d39"
format_stamp: "Formatted at 2026-09-24 03:12:39 on dist-test-slave-jtnz"
server_key: "79409122309c714283a71632d85eb323"
server_key_iv: "2248e6e15197d957f3bca67ad5b0060e"
server_key_version: "encryptionkey@0"
I20260924 03:12:39.366693 22469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:39.367359 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:39.367542 22469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:39.367630 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "5f35d6f5a16c4c3f9d01639750674d39"
format_stamp: "Formatted at 2026-09-24 03:12:39 on dist-test-slave-jtnz"
server_key: "79409122309c714283a71632d85eb323"
server_key_iv: "2248e6e15197d957f3bca67ad5b0060e"
server_key_version: "encryptionkey@0"
I20260924 03:12:39.367710 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:39.370796 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:39.372843 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:39.373015 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:39.376664 22469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36785
I20260924 03:12:39.376689 22536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36785 every 8 connection(s)
I20260924 03:12:39.376962 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:39.377755 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:39.377892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22469
I20260924 03:12:39.378036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:39.378261 392 external_mini_cluster.cc:1428] Setting key 536abb081ab65b68a98d3c18f2749909
I20260924 03:12:39.380493 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: Bootstrap starting.
I20260924 03:12:39.380667 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:39.378715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60877 (local address 127.0.98.62:36785)
0924 03:12:39.378905 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:39.378920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:39.378943 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:12:39.379303 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:39.379309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:39.379330 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:39.379446 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:39.380181 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:39.380184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:39.380184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:39.380187 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:39.380219 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:39.380228 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:39.380283 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:39.380375 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:12:39.380402 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260924 03:12:39.381209 22537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:39.381544 22537 log.cc:824] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:39.381572 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:39.382249 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: No bootstrap required, opened a new log
I20260924 03:12:39.383858 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:39.382982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57063 (local address 127.0.98.62:36785)
0924 03:12:39.383017 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:39.383018 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:39.383080 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:12:39.383142 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:39.383142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:39.383145 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:39.383188 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:39.383662 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:39.383662 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:39.383663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:12:39.383666 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:39.383675 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:12:39.383678 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:39.383715 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:39.383774 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:12:39.383783 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:39.384748 22537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 03:12:39.384886 22537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:12:39.384930 22537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f35d6f5a16c4c3f9d01639750674d39, State: Initialized, Role: FOLLOWER
I20260924 03:12:39.385040 22537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 03:12:39.385138 22537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:12:39.385180 22537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:12:39.385229 22537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:39.385720 22537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 03:12:39.385828 22537 leader_election.cc:302] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f35d6f5a16c4c3f9d01639750674d39; no voters:
I20260924 03:12:39.385990 22537 leader_election.cc:288] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:12:39.386013 22547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:39.386173 22547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [term 1 LEADER]: Becoming Leader. State: Replica: 5f35d6f5a16c4c3f9d01639750674d39, State: Running, Role: LEADER
I20260924 03:12:39.386235 22537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:39.386332 22547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 03:12:39.386729 22548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f35d6f5a16c4c3f9d01639750674d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } }
I20260924 03:12:39.386817 22548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:39.387001 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f35d6f5a16c4c3f9d01639750674d39. Latest consensus state: current_term: 1 leader_uuid: "5f35d6f5a16c4c3f9d01639750674d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f35d6f5a16c4c3f9d01639750674d39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } }
I20260924 03:12:39.387096 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39 [sys.catalog]: This master's current role is: LEADER
W20260924 03:12:39.388024 22563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:39.388065 22563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:39.388126 22556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:39.388276 22556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:39.389567 22556 catalog_manager.cc:1347] Generated new cluster ID: 2ff0a5447f294771ad52c07cccf2a40e
I20260924 03:12:39.389621 22556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:39.416075 22556 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:39.416231 22556 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:39.426559 22556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f35d6f5a16c4c3f9d01639750674d39: Generated new TSK 0
I20260924 03:12:39.426710 22556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:12:39.429813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22469
2026-09-24T03:12:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1215 ms)
[----------] 18 tests from AuthTokenIssuingTest (21819 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:39 dist-test-slave-jtnz krb5kdc[22568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:39 dist-test-slave-jtnz krb5kdc[22568](info): set up 2 sockets
Sep 24 03:12:39 dist-test-slave-jtnz krb5kdc[22568](info): commencing operation
krb5kdc: starting...
W20260924 03:12:40.385952 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:40.490911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:40.565747 22581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:40.565960 22581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:40.565989 22581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:40.566028 22581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:40.567166 22581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:40.567229 22581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:40.567252 22581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:40.567287 22581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:40.567312 22581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:40.568743 22581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:40.568997 22581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:40.569199 22581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:40.573416 22586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.573490 22581 server_base.cc:1034] running on GCE node
W20260924 03:12:40.573536 22587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:40.573536 22589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.573876 22581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:40.574142 22581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:40.575316 22581 hybrid_clock.cc:648] HybridClock initialized: now 1790219560575302 us; error 31 us; skew 500 ppm
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:40.577342 22581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:40.577749 22581 webserver.cc:460] Webserver started at http://127.0.98.62:33995/ using document root <none> and password file <none>
I20260924 03:12:40.577956 22581 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:40.578017 22581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:40.578152 22581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:40.579049 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "1d2cae93360549a8babccc2d1cd6de74"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "bc5af5bdc618f3ad61eb4dde863e5da3"
server_key_iv: "78899b77bd69acf370871bcf15fd0d6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.579391 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d2cae93360549a8babccc2d1cd6de74"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "bc5af5bdc618f3ad61eb4dde863e5da3"
server_key_iv: "78899b77bd69acf370871bcf15fd0d6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.580570 22581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:40.581175 22596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:40.581315 22581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:12:40.581394 22581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "1d2cae93360549a8babccc2d1cd6de74"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "bc5af5bdc618f3ad61eb4dde863e5da3"
server_key_iv: "78899b77bd69acf370871bcf15fd0d6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.581468 22581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:40.592258 22581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:40.593967 22581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:40.594115 22581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:40.597630 22581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44869
I20260924 03:12:40.597678 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44869 every 8 connection(s)
I20260924 03:12:40.597972 22581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:40.598884 22649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:40.600185 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.604660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22581
I20260924 03:12:40.604759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:40.604995 392 external_mini_cluster.cc:1428] Setting key 9670df97ec32d9874bc167f4ac147789
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:40.609586 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.600827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.62:44869)
0924 03:12:40.600993 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.601002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.601131 (+ 129us) server_negotiation.cc:407] Connection header received
0924 03:12:40.601664 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.601668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.601691 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.601787 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.602347 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.603148 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.604054 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.604344 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.604472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.606733 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.606739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.606739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.606745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.608176 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.608545 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.608547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.608547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.608622 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.608820 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.608821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.608821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.609001 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.609083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.609249 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.609379 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260924 03:12:40.609587 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.600738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:42652)
0924 03:12:40.600984 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.601024 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.601517 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.601794 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.601796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.602055 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.602255 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.602260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.603254 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.603255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.603956 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.603958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.604461 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.604462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.604527 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.605052 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.605062 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.606650 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.608280 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.608286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.608287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.608462 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.608684 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.608685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.608685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.608759 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.609066 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.609070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.609125 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.609227 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.609422 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":88,"thread_start_us":110,"threads_started":1}
I20260924 03:12:40.610479 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.610796 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.610682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53642)
0924 03:12:40.610728 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.610767 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:12:40.610957 22597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
I20260924 03:12:40.611357 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.611264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53656)
0924 03:12:40.611318 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.611339 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:12:40.611456 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:12:40.628307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36191
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:40.660743 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } attempt: 1
I20260924 03:12:40.661233 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.661073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53664)
0924 03:12:40.661155 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.661199 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 03:12:40.661616 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.661513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53676)
0924 03:12:40.661569 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.661596 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:12:40.661738 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
W20260924 03:12:40.700670 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:40.700867 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:40.700884 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:40.700922 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:40.702054 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:40.702091 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:40.702104 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:40.702139 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:40.702165 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:40.703673 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:40.703912 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:40.704128 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:40.708559 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:40.708541 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.708580 22656 server_base.cc:1034] running on GCE node
W20260924 03:12:40.708540 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.708912 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:40.709177 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:40.710330 22656 hybrid_clock.cc:648] HybridClock initialized: now 1790219560710316 us; error 25 us; skew 500 ppm
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:40.712491 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:12:40.712909 22656 webserver.cc:460] Webserver started at http://127.0.98.61:41465/ using document root <none> and password file <none>
I20260924 03:12:40.713128 22656 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:40.713191 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:40.713331 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:40.714380 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data/instance:
uuid: "cbf648aa1b9e435cb92c4def86b4a4e7"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "d62655097f8575545fc265e53d682f04"
server_key_iv: "4b774359f144e9993729292d19de0eda"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.714711 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal/instance:
uuid: "cbf648aa1b9e435cb92c4def86b4a4e7"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "d62655097f8575545fc265e53d682f04"
server_key_iv: "4b774359f144e9993729292d19de0eda"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.715884 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:40.716527 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:40.716672 22656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:40.716759 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "cbf648aa1b9e435cb92c4def86b4a4e7"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "d62655097f8575545fc265e53d682f04"
server_key_iv: "4b774359f144e9993729292d19de0eda"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.716827 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:40.729948 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } attempt: 2
I20260924 03:12:40.730319 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.730168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53692)
0924 03:12:40.730250 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.730286 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 03:12:40.730684 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.730561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53704)
0924 03:12:40.730627 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.730656 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:12:40.730777 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36191: connect: Connection refused (error 111)
I20260924 03:12:40.731758 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:40.733165 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:40.733270 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:40.737083 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36191
I20260924 03:12:40.737128 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36191 every 8 connection(s)
I20260924 03:12:40.737403 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:12:40.738319 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:40.739531 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.742166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22656
I20260924 03:12:40.742302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-1/wal/instance
I20260924 03:12:40.742484 392 external_mini_cluster.cc:1428] Setting key fc0c7f2355af5f7e75e84fcf1742052e
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:40.747824 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.740016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.98.62:44869)
0924 03:12:40.740048 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.740049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.740211 (+ 162us) server_negotiation.cc:407] Connection header received
0924 03:12:40.740581 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.740582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.740591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.740635 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.741144 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.741584 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.742594 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.742895 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.742967 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.745096 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.745097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.745097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.745103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.746535 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.746904 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.746905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.746906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.746966 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.747224 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.747224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.747225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.747331 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.747398 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.747661 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.747747 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:40.748000 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.739981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:42654)
0924 03:12:40.740106 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.740139 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.740516 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.740676 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.740678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.740896 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.741061 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.741064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.741657 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.741658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.742503 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.742505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.742985 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.742986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.743076 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.743560 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.743567 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.744994 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.746617 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.746623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.746625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.746822 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.747034 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.747034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.747035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.747168 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.747391 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.747397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.747521 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.747630 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.747768 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260924 03:12:40.748593 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:40.756814 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.748810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53714)
0924 03:12:40.748851 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.748861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.748891 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.749130 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.749131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.749252 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.749288 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.749289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.750267 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.750267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.750806 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.750807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.751213 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.751214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.751285 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.751692 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.751696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.753818 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.755911 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.755912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.755913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.756062 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.756278 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.756279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.756279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.756324 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.756650 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.756651 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.756678 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.756717 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.756768 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:12:40.756942 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.748888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.0.98.61:36191)
0924 03:12:40.749019 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.749022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.749036 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:12:40.749050 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.749051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.749060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.749130 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.749399 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.750138 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.750871 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.751140 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.751190 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.753894 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.753901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.753902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.753909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.755833 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.756142 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.756145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.756146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.756209 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.756385 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.756386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.756386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.756570 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.756659 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.756750 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.756823 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 03:12:40.757409 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.757680 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.757584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36206)
0924 03:12:40.757616 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.757659 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 03:12:40.757815 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
I20260924 03:12:40.758244 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.758118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36212)
0924 03:12:40.758172 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.758215 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 03:12:40.758378 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33757: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:12:40.766460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33757
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:40.807600 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } attempt: 1
I20260924 03:12:40.808013 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.807878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36224)
0924 03:12:40.807953 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.807991 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:12:40.808390 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.808307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36226)
0924 03:12:40.808351 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.808374 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:12:40.808490 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33757: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
I20260924 03:12:40.821938 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } attempt: 3
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:40.828604 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.822280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.0.98.61:36191)
0924 03:12:40.822318 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.822320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.822332 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:12:40.822427 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.822428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.822439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.822510 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.822799 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.823273 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.824018 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.824228 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.824302 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.826273 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.826274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.826275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.826286 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.827545 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.827868 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.827869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.827869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.827923 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.828126 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.828127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.828127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.828230 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.828302 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.828460 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.828530 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:12:40.828603 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.822252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53716)
0924 03:12:40.822290 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.822301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.822350 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.822514 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.822515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.822668 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.822723 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.822724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.823372 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.823373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.823915 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.823916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.824299 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.824301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.824381 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.824851 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.824855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.826148 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.827636 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.827637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.827638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.827800 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.828000 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.828001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.828001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.828065 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.828295 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.828296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.828339 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.828419 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.828514 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 03:12:40.829141 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.829450 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.829362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36232)
0924 03:12:40.829404 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.829429 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:12:40.829738 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.829663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36248)
0924 03:12:40.829701 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.829721 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 03:12:40.829819 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33757: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
W20260924 03:12:40.841082 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:40.841248 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:40.841264 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:40.841325 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:40.842422 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:40.842458 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:40.842471 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:40.842512 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:40.842540 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:40.843889 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869,127.0.98.61:36191,127.0.98.60:33757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:40.844123 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:40.844331 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:40.848512 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:40.848588 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.848548 22731 server_base.cc:1034] running on GCE node
W20260924 03:12:40.848686 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:40.848893 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:40.849174 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:40.850303 22731 hybrid_clock.cc:648] HybridClock initialized: now 1790219560850290 us; error 24 us; skew 500 ppm
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:40.852507 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:12:40.852928 22731 webserver.cc:460] Webserver started at http://127.0.98.60:46439/ using document root <none> and password file <none>
I20260924 03:12:40.853124 22731 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:40.853164 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:40.853294 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:40.854321 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data/instance:
uuid: "a687fcce63104b7db35488d90171bd94"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "908046134d2f9c0ae7803aacda336db6"
server_key_iv: "87cd9d77c6d20332c8d4a96f23c779cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.854664 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal/instance:
uuid: "a687fcce63104b7db35488d90171bd94"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "908046134d2f9c0ae7803aacda336db6"
server_key_iv: "87cd9d77c6d20332c8d4a96f23c779cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.855901 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:12:40.856559 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:40.856738 22731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:40.856822 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "a687fcce63104b7db35488d90171bd94"
format_stamp: "Formatted at 2026-09-24 03:12:40 on dist-test-slave-jtnz"
server_key: "908046134d2f9c0ae7803aacda336db6"
server_key_iv: "87cd9d77c6d20332c8d4a96f23c779cf"
server_key_version: "encryptionkey@0"
I20260924 03:12:40.856889 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:40.860937 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } attempt: 1
I20260924 03:12:40.861245 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.861136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36254)
0924 03:12:40.861186 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.861225 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:12:40.861529 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.861437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36270)
0924 03:12:40.861482 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.861505 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:12:40.861611 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33757: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
I20260924 03:12:40.872579 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:40.874616 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:40.874756 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:40.876602 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } attempt: 2
I20260924 03:12:40.876894 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.876804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36276)
0924 03:12:40.876838 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.876866 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 03:12:40.877280 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.877164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36282)
0924 03:12:40.877215 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.877246 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 03:12:40.877363 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33757: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33757: connect: Connection refused (error 111)
I20260924 03:12:40.878573 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33757
I20260924 03:12:40.878597 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33757 every 8 connection(s)
I20260924 03:12:40.878870 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:12:40.879684 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:40.880048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22731
I20260924 03:12:40.880128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219310260848-392-0/minicluster-data/master-2/wal/instance
I20260924 03:12:40.880294 392 external_mini_cluster.cc:1428] Setting key baaa6c396705b620cdaa1086f019479c
I20260924 03:12:40.881352 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:12:40.889361 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.882092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.98.62:44869)
0924 03:12:40.882146 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.882148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.882292 (+ 144us) server_negotiation.cc:407] Connection header received
0924 03:12:40.882359 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.882359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.882367 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.882403 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.882896 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.883333 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.884183 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.884425 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.884473 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.886371 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.886371 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.886372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.886377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.887965 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.888405 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.888406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.888406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.888458 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.888709 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.888710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.888710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.888830 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.888902 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.889225 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.889278 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:12:40.889529 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.882021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:42656)
0924 03:12:40.882217 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.882252 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.882305 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.882424 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.882426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.882693 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.882809 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.882814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.883423 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.883424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.884117 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.884119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.884513 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.884514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.884627 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.885031 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.885038 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.886298 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.888039 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.888043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.888045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.888337 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.888521 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.888522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.888522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.888643 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.888915 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.888920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.889062 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.889180 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.889318 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260924 03:12:40.889938 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.880699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.0.98.60:33757)
0924 03:12:40.880881 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.880888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.880909 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:12:40.881253 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.881258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.881295 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.881398 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.882240 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.883398 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.884219 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.884627 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.884688 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.886859 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.886865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.886866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.886873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.888567 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.888923 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.888926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.888926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.888986 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.889146 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.889146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.889147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.889278 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.889392 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.889772 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.889849 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260924 03:12:40.889973 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:40.896390 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.890300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.0.98.61:36191)
0924 03:12:40.890344 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.890345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.890355 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:12:40.890376 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.890377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.890385 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.890437 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.890775 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.891432 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.892174 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.892471 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.892538 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.894138 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.894139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.894139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.894146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.895407 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.895747 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.895747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.895748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.895792 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.895977 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.895978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.895978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.896100 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.896158 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.896296 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.896344 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:12:40.896432 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.890250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36191 (local address 127.0.0.1:53726)
0924 03:12:40.890279 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.890288 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.890321 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.890440 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.890441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.890653 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.890691 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.890693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.891498 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.891499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.892105 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.892107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.892541 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.892541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.892599 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.892908 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.892913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.894049 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.895488 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.895489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.895489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.895677 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.895849 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.895850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.895851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.895910 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.896164 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.896165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.896207 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.896281 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.896358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 03:12:40.896780 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:40.897867 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.890625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.98.62:44869)
0924 03:12:40.890657 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.890658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.890665 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:40.890707 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.890708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.890717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.890771 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.891270 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.891827 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.892617 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.892885 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.892957 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.895313 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.895314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.895314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.895320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.896807 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.897166 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.897167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.897167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.897223 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.897404 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.897405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.897405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.897526 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.897676 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.897708 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.897788 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:40.903738 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.897004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36304)
0924 03:12:40.897062 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.897070 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.897107 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.897224 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.897225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.897367 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.897399 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.897401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.898137 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.898137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.898918 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.898920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.899255 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.899256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.899301 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.899675 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.899680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.900891 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.902601 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.902602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.902602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.902753 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.903063 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.903064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.903064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.903108 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.903355 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.903358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.903439 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.903517 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.903611 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 03:12:40.903776 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.897032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.0.98.60:33757)
0924 03:12:40.897056 (+ 24us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.897057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.897096 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:12:40.897168 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.897169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.897177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.897223 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.897462 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.898071 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.898986 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.899175 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.899245 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.900986 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.900987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.900987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.900992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.902530 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.902840 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.902840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.902841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.902889 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.903170 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.903171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.903171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.903281 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.903340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.903544 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.903670 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":11}
I20260924 03:12:40.905027 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.898412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.0.98.61:36191)
0924 03:12:40.898441 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.898443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.898450 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:40.898467 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.898467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.898474 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.898521 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.899048 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.899469 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.900105 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.900271 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.900340 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.902662 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.902663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.902663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.902681 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.904078 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.904434 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.904435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.904435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.904483 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.904671 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.904671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.904671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.904747 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.904830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.904927 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.904981 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:40.905269 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: Bootstrap starting.
I20260924 03:12:40.905889 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:40.906090 22799 log.cc:824] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:40.906816 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: No bootstrap required, opened a new log
I20260924 03:12:40.907392 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:40.908298 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.908449 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a687fcce63104b7db35488d90171bd94, State: Initialized, Role: FOLLOWER
I20260924 03:12:40.908524 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.908771 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.908811 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:40.908876 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.908390 22623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d2cae93360549a8babccc2d1cd6de74"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672
W20260924 03:12:40.910561 22816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:40.910643 22816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:40.910808 22731 master_runner.cc:186] Error getting master registration for 127.0.98.62:44869: OK, instance_id { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" instance_seqno: 1790219560596641 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:40.913298 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:12:40.936754 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } attempt: 2
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:12:40.943120 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.936960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36310)
0924 03:12:40.937007 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.937017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.937051 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.937197 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.937197 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.937343 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.937371 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.937373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.937965 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.937966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.938516 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.938518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.938848 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.938849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.938901 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.939309 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.939314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.940457 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.942122 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.942123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.942123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.942369 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.942551 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.942551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.942552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.942604 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.942856 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.942857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.942892 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.942958 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.943033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:12:40.943140 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.937044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.0.98.60:33757)
0924 03:12:40.937094 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.937095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.937102 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:40.937125 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.937125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.937132 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.937193 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.937462 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.937871 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.938595 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.938767 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.938846 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.940533 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.940534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.940534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.940542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.942043 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.942436 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.942437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.942437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.942488 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.942660 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.942661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.942661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.942793 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.942855 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.942997 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.943063 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:12:40.944689 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: Bootstrap starting.
I20260924 03:12:40.945327 22649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:40.945611 22649 log.cc:824] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:40.946432 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: No bootstrap required, opened a new log
I20260924 03:12:40.947669 22649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.947791 22649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2cae93360549a8babccc2d1cd6de74, State: Initialized, Role: FOLLOWER
I20260924 03:12:40.947863 22649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.948135 22649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:40.948160 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:12:40.948112 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.948240 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:40.948257 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.949012 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.949357 22623 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [CANDIDATE]: Term 1 election: Requested vote from peers cbf648aa1b9e435cb92c4def86b4a4e7 (127.0.98.61:36191), a687fcce63104b7db35488d90171bd94 (127.0.98.60:33757)
W20260924 03:12:40.949927 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:40.949973 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:40.950888 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d2cae93360549a8babccc2d1cd6de74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a687fcce63104b7db35488d90171bd94"
I20260924 03:12:40.951035 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:40.951817 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2cae93360549a8babccc2d1cd6de74 in term 1.
I20260924 03:12:40.952134 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2cae93360549a8babccc2d1cd6de74, a687fcce63104b7db35488d90171bd94; no voters:
I20260924 03:12:40.950623 22581 master_runner.cc:186] Error getting master registration for 127.0.98.61:36191: OK, instance_id { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" instance_seqno: 1790219560736027 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:40.953431 22581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:12:40.953488 22819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:40.953606 22819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2cae93360549a8babccc2d1cd6de74, State: Running, Role: LEADER
I20260924 03:12:40.953673 22819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.954051 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2cae93360549a8babccc2d1cd6de74. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.954130 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:40.954710 22841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:40.955323 22841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:40.957907 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [term 1 FOLLOWER]: Refusing update from remote peer 1d2cae93360549a8babccc2d1cd6de74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:40.958276 22819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:40.958652 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.950686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.98.62:44869)
0924 03:12:40.950715 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.950717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.950824 (+ 107us) server_negotiation.cc:407] Connection header received
0924 03:12:40.950958 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.950959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.950969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.951002 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.951784 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.952442 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.953431 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.953705 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.953748 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.955346 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.955347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.955347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.955353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.957141 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.957818 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.957819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.957820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.957873 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.958091 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.958092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.958093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.958228 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.958292 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.958496 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.958569 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:40.959501 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2cae93360549a8babccc2d1cd6de74. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.959578 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.960067 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.950696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.0.98.61:36191)
0924 03:12:40.950728 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.950729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.952626 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 03:12:40.952715 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.952715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.952724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.952751 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.953499 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.954220 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.955191 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.955886 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.955928 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.957223 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.957224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.957224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.957231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.958868 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.959230 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.959231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.959231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.959282 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.959520 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.959520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.959521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.959626 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.959700 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.959902 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.959984 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:40.960376 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.960448 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:40.960654 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.960716 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.961189 22843 mvcc.cc:205] Tried to move back new op lower bound from 7332739321680809984 to 7332739321666801664. Current Snapshot: MvccSnapshot[applied={T|T < 7332739321680809984 or (T in {7332739321680809984})}]
I20260924 03:12:40.961341 22842 mvcc.cc:205] Tried to move back new op lower bound from 7332739321680809984 to 7332739321666801664. Current Snapshot: MvccSnapshot[applied={T|T < 7332739321680809984 or (T in {7332739321680809984})}]
I20260924 03:12:40.966848 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.952356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.60:33757)
0924 03:12:40.952386 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.952387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.958315 (+ 5928us) server_negotiation.cc:407] Connection header received
0924 03:12:40.958408 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.958409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.958419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.958461 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.959031 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.959675 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.960567 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.960838 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.960944 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.964130 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.964131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.964132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.964138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.965660 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.966033 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.966034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.966034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.966089 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.966302 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.966302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.966303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.966431 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.966514 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.966702 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.966769 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:40.968475 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } attempt: 3
I20260924 03:12:40.969089 22841 catalog_manager.cc:1347] Generated new cluster ID: 5c696170ccc0475093e37c81a1564954
I20260924 03:12:40.969141 22841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:12:40 dist-test-slave-jtnz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:12:40.977742 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.968693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33757 (local address 127.0.0.1:36324)
0924 03:12:40.968724 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:40.968745 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:40.968783 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:40.968941 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:40.968942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:40.969092 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:40.969131 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.969133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.970046 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.970047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.970838 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:40.970841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.971968 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.971969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:40.972035 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.972451 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:40.972456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:40.973728 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:40.976669 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.976670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.976671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.976871 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.977128 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:40.977128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:40.977129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:40.977190 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:40.977490 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:40.977491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:40.977537 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:12:40.977592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:12:40.977663 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 03:12:40.978142 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:40.968731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.98.60:33757)
0924 03:12:40.968771 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:40.968772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:40.968780 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:12:40.968837 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:40.968838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:40.968846 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:40.968904 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:40.969227 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.969930 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.971394 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:40.971859 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:40.972941 (+ 1082us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:40.973826 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:40.973827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:40.973827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:40.973833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:40.976557 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.976956 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.976957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.976958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.977036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:40.977231 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:40.977231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:40.977232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:40.977410 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:40.977498 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:40.977627 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:12:40.977734 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:12:40.979483 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: Bootstrap starting.
I20260924 03:12:40.980068 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:40.980306 22724 log.cc:824] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:40.981076 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: No bootstrap required, opened a new log
I20260924 03:12:40.982784 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.982931 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf648aa1b9e435cb92c4def86b4a4e7, State: Initialized, Role: FOLLOWER
I20260924 03:12:40.983006 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } }
I20260924 03:12:40.983299 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:40.983387 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:12:40.983769 22677 tablet.cc:2369] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:12:40.984226 22697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [term 1 FOLLOWER]: Refusing update from remote peer 1d2cae93360549a8babccc2d1cd6de74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:40.984376 22698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d2cae93360549a8babccc2d1cd6de74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7"
I20260924 03:12:40.985314 22698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2cae93360549a8babccc2d1cd6de74 in term 1.
I20260924 03:12:40.985682 22839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:40.985756 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.985940 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.987309 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2cae93360549a8babccc2d1cd6de74. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.987581 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:40.987699 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.987795 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:12:40.990847 22860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:40.990897 22860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:40.991516 22861 mvcc.cc:205] Tried to move back new op lower bound from 7332739321680809984 to 7332739321666801664. Current Snapshot: MvccSnapshot[applied={T|T < 7332739321680809984}]
I20260924 03:12:40.997466 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d2cae93360549a8babccc2d1cd6de74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2cae93360549a8babccc2d1cd6de74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } peers { permanent_uuid: "cbf648aa1b9e435cb92c4def86b4a4e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36191 } } peers { permanent_uuid: "a687fcce63104b7db35488d90171bd94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33757 } } }
I20260924 03:12:40.997551 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:40.998314 22841 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:40.998967 22841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:41.006254 22841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2cae93360549a8babccc2d1cd6de74: Generated new TSK 0
I20260924 03:12:41.006423 22841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:12:41.033594 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.031983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.0.98.61:36191)
0924 03:12:41.032019 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.032021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.032774 (+ 753us) server_negotiation.cc:407] Connection header received
0924 03:12:41.033081 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.033082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.033091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.033121 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.033535 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53762: BlockingRecv error: recv got EOF from 127.0.0.1:53762 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:12:41.034153 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.032169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.98.62:44869)
0924 03:12:41.032201 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.032203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.032854 (+ 651us) server_negotiation.cc:407] Connection header received
0924 03:12:41.033734 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.033735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.033745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.033774 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.034099 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42688: BlockingRecv error: recv got EOF from 127.0.0.1:42688 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:12:41.034785 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.032341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.0.98.60:33757)
0924 03:12:41.032373 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.032374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.032989 (+ 615us) server_negotiation.cc:407] Connection header received
0924 03:12:41.034357 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.034358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.034367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.034396 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.034733 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36338: BlockingRecv error: recv got EOF from 127.0.0.1:36338 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:12:41.036859 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.036095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.0.98.61:36191)
0924 03:12:41.036139 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.036140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.036403 (+ 263us) server_negotiation.cc:407] Connection header received
0924 03:12:41.036518 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.036519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.036534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.036561 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.036819 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53768: BlockingRecv error: recv got EOF from 127.0.0.1:53768 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 03:12:41.039669 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.038946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.98.61:36191)
0924 03:12:41.038977 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.038978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.039222 (+ 244us) server_negotiation.cc:407] Connection header received
0924 03:12:41.039309 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.039310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.039319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.039346 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.039630 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53778: BlockingRecv error: recv got EOF from 127.0.0.1:53778 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:12:41.043342 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.042727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.0.98.61:36191)
0924 03:12:41.042755 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.042756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.042920 (+ 164us) server_negotiation.cc:407] Connection header received
0924 03:12:41.043003 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.043004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.043012 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.043039 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.043307 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53784: BlockingRecv error: recv got EOF from 127.0.0.1:53784 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 03:12:41.049167 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.048469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.61:36191)
0924 03:12:41.048498 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.048499 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.048707 (+ 208us) server_negotiation.cc:407] Connection header received
0924 03:12:41.048825 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.048826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.048834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.048860 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.049130 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53788: BlockingRecv error: recv got EOF from 127.0.0.1:53788 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:12:41.058885 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.058183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.98.61:36191)
0924 03:12:41.058235 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.058237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.058440 (+ 203us) server_negotiation.cc:407] Connection header received
0924 03:12:41.058527 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.058528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.058537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.058563 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.058847 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53796: BlockingRecv error: recv got EOF from 127.0.0.1:53796 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 03:12:41.076680 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.075941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.98.61:36191)
0924 03:12:41.075970 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.075972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.076171 (+ 199us) server_negotiation.cc:407] Connection header received
0924 03:12:41.076258 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.076259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.076268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.076295 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.076631 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53798: BlockingRecv error: recv got EOF from 127.0.0.1:53798 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:12:41.110659 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.109862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.98.61:36191)
0924 03:12:41.109910 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.109912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.110043 (+ 131us) server_negotiation.cc:407] Connection header received
0924 03:12:41.110093 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.110094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.110105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.110169 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.110610 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53806: BlockingRecv error: recv got EOF from 127.0.0.1:53806 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 03:12:41.176290 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.175684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.0.98.61:36191)
0924 03:12:41.175765 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.175767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.175845 (+ 78us) server_negotiation.cc:407] Connection header received
0924 03:12:41.175889 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.175890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.175899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.175947 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.176240 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53816: BlockingRecv error: recv got EOF from 127.0.0.1:53816 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 03:12:41.306020 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.305306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.61:36191)
0924 03:12:41.305377 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.305379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.305504 (+ 125us) server_negotiation.cc:407] Connection header received
0924 03:12:41.305563 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.305564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.305574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.305615 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.305933 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53832: BlockingRecv error: recv got EOF from 127.0.0.1:53832 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 03:12:41.563845 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:41.563076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.0.98.61:36191)
0924 03:12:41.563137 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:41.563139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:41.563171 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:12:41.563238 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:41.563239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:41.563250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:41.563297 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:41.563760 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53840: BlockingRecv error: recv got EOF from 127.0.0.1:53840 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:12:41.911016 22816 catalog_manager.cc:1259] Loaded cluster ID: 5c696170ccc0475093e37c81a1564954
I20260924 03:12:41.911092 22816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: loading cluster ID for follower catalog manager: success
I20260924 03:12:41.911844 22816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: acquiring CA information for follower catalog manager: success
I20260924 03:12:41.912141 22816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a687fcce63104b7db35488d90171bd94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:41.991267 22860 catalog_manager.cc:1259] Loaded cluster ID: 5c696170ccc0475093e37c81a1564954
I20260924 03:12:41.991318 22860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: loading cluster ID for follower catalog manager: success
I20260924 03:12:41.991981 22860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: acquiring CA information for follower catalog manager: success
I20260924 03:12:41.992204 22860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbf648aa1b9e435cb92c4def86b4a4e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:42.079020 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.076839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.98.61:36191)
0924 03:12:42.076917 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:42.076918 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:42.076982 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:12:42.077009 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:42.077010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:42.077047 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:42.077118 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:42.077239 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.077656 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.078379 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.078562 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.078612 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:42.078736 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:42.078836 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:42.078890 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:42.078937 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 03:12:42.078959 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":37,"threads_started":1}
I20260924 03:12:42.082806 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.080776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.98.60:33757)
0924 03:12:42.080889 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:42.080890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:42.080898 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:12:42.080958 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:42.080958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:42.080987 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:42.081046 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:42.081124 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.081565 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.082210 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.082377 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.082424 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:42.082527 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:42.082597 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:42.082638 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:42.082717 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:12:42.082738 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260924 03:12:42.086964 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.084321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.0.98.60:33757)
0924 03:12:42.084355 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:42.084357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:42.084604 (+ 247us) server_negotiation.cc:407] Connection header received
0924 03:12:42.084655 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:42.084656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:42.084668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:42.084706 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:42.084904 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.085577 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.086367 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.086604 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.086674 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:42.086745 (+ 71us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:42.086769 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:42.086848 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:42.086877 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 03:12:42.086895 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:12:42.087432 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.084348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.0.98.61:36191)
0924 03:12:42.084473 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:42.084475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:42.084527 (+ 52us) server_negotiation.cc:407] Connection header received
0924 03:12:42.084622 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:42.084623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:42.084632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:42.084659 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:42.084818 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.085518 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.086550 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.086864 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.086927 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:42.087090 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:42.087117 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:42.087141 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:42.087222 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:12:42.087247 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 03:12:42.087920 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:42.084262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.98.62:44869)
0924 03:12:42.084527 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:42.084530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:42.084562 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:12:42.084674 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:42.084676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:42.084734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:42.084801 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:42.084956 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.085439 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.086961 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:42.087240 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:42.087323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:42.087571 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:42.087666 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:42.087753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:42.087813 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 03:12:42.087838 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":47,"threads_started":1}
I20260924 03:12:42.092592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22581
I20260924 03:12:42.097388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22656
I20260924 03:12:42.102074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22731
2026-09-24T03:12:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2674 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:12:42 dist-test-slave-jtnz krb5kdc[22947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:12:42 dist-test-slave-jtnz krb5kdc[22947](info): set up 2 sockets
Sep 24 03:12:42 dist-test-slave-jtnz krb5kdc[22947](info): commencing operation
krb5kdc: starting...
W20260924 03:12:43.097965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/test-user.keytab.
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:12:43.208807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:12:43.286278 22960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:43.286531 22960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:43.286561 22960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:43.286602 22960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:43.287693 22960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:43.287765 22960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:43.287788 22960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:43.287809 22960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:43.287837 22960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:43.289274 22960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:43.289542 22960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:43.289755 22960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:43.294303 22968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:43.294291 22966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:43.294414 22960 server_base.cc:1034] running on GCE node
W20260924 03:12:43.294281 22965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:43.294788 22960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:43.295056 22960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:43.296224 22960 hybrid_clock.cc:648] HybridClock initialized: now 1790219563296220 us; error 35 us; skew 500 ppm
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:43.298602 22960 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:12:43.299019 22960 webserver.cc:460] Webserver started at http://127.0.98.62:41855/ using document root <none> and password file <none>
I20260924 03:12:43.299234 22960 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:43.299300 22960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:43.299427 22960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:43.300413 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data/instance:
uuid: "9e6fa5c9ae324f2e914f6adfbc724280"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3af068ac7980faf201c10aca190b2346"
server_key_iv: "9a0db35d0b337bdcdcf63099308475c9"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.300767 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e6fa5c9ae324f2e914f6adfbc724280"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3af068ac7980faf201c10aca190b2346"
server_key_iv: "9a0db35d0b337bdcdcf63099308475c9"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.301998 22960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:12:43.302759 22975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:43.302917 22960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:43.303014 22960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal
uuid: "9e6fa5c9ae324f2e914f6adfbc724280"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3af068ac7980faf201c10aca190b2346"
server_key_iv: "9a0db35d0b337bdcdcf63099308475c9"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.303099 22960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:43.320175 22960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:43.322474 22960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:43.322643 22960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:43.326133 22960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45011
I20260924 03:12:43.326164 23027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45011 every 8 connection(s)
I20260924 03:12:43.326494 22960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:12:43.327353 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:43.328761 23028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.332723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 22960
I20260924 03:12:43.332826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-0/wal/instance
I20260924 03:12:43.333000 392 external_mini_cluster.cc:1428] Setting key 10da428653aad0d82beb20e03321096c
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:43.338377 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.329497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.0.98.62:45011)
0924 03:12:43.329614 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.329618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.329655 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:12:43.330346 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.330349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.330387 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.330495 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.331062 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.331831 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.332621 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.333114 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.333225 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.335270 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.335276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.335276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.335282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.336776 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.337254 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.337256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.337256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.337325 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.337577 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.337578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.337578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.337749 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.337809 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.338034 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.338143 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260924 03:12:43.338403 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.329343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45011 (local address 127.0.0.1:59360)
0924 03:12:43.329486 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.329516 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.330148 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.330529 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.330531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.330835 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.330981 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.330986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.331908 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.331908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.332546 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.332547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.333211 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.333212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.333266 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.333738 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.333745 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.335179 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.336868 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.336876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.336878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.337173 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.337394 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.337395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.337396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.337509 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.337819 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.337826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.337911 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.338018 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.338262 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":69,"thread_start_us":61,"threads_started":1}
I20260924 03:12:43.339167 23028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.339489 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.339373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54202)
0924 03:12:43.339423 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.339467 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:12:43.339636 22976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
I20260924 03:12:43.340036 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.339917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54204)
0924 03:12:43.339976 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.340007 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 03:12:43.340144 23028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:12:43.355139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46383
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:43.389364 23028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } attempt: 1
I20260924 03:12:43.389793 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.389669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54216)
0924 03:12:43.389724 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.389763 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:12:43.390136 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.390045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54226)
0924 03:12:43.390090 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.390115 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:12:43.390311 23028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
W20260924 03:12:43.437062 23035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:43.437244 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:43.437260 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:43.437279 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:43.438509 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:43.438553 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:43.438566 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:43.438578 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:43.438591 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:43.440016 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:43.440251 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:43.440471 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:12:43.445063 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:43.445070 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:43.445216 23035 server_base.cc:1034] running on GCE node
W20260924 03:12:43.445101 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:43.445501 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:43.445819 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:43.446979 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790219563446966 us; error 24 us; skew 500 ppm
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:43.449306 23035 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:12:43.449779 23035 webserver.cc:460] Webserver started at http://127.0.98.61:37777/ using document root <none> and password file <none>
I20260924 03:12:43.450001 23035 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:43.450068 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:43.450256 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:43.451251 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data/instance:
uuid: "48a7af97fb0a4a48abb6b1bced28e788"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "39597be10e9c233203241edaacb889fb"
server_key_iv: "761d3951ef8dae099113b717cea23a6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.451606 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal/instance:
uuid: "48a7af97fb0a4a48abb6b1bced28e788"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "39597be10e9c233203241edaacb889fb"
server_key_iv: "761d3951ef8dae099113b717cea23a6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.452880 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:12:43.453560 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:43.453739 23035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:12:43.453833 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal
uuid: "48a7af97fb0a4a48abb6b1bced28e788"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "39597be10e9c233203241edaacb889fb"
server_key_iv: "761d3951ef8dae099113b717cea23a6e"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.453913 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:43.458455 23028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } attempt: 2
I20260924 03:12:43.458811 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.458688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54230)
0924 03:12:43.458744 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.458778 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:12:43.459141 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.459052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54242)
0924 03:12:43.459095 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.459118 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:12:43.459223 23028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46383: connect: Connection refused (error 111)
I20260924 03:12:43.472421 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:43.474133 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:43.474310 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:43.478093 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46383
I20260924 03:12:43.478127 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46383 every 8 connection(s)
I20260924 03:12:43.478485 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:12:43.479094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 23035
I20260924 03:12:43.479228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-1/wal/instance
I20260924 03:12:43.479465 392 external_mini_cluster.cc:1428] Setting key 137351cb24b60918290e34f08692a3d1
I20260924 03:12:43.479744 23103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:43.481707 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:43.493570 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.482518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.0.98.62:45011)
0924 03:12:43.482589 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.482591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.482822 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:12:43.483513 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.483514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.483525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.483567 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.484421 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.484880 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.486394 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.486578 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.486631 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.490839 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.490840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.490840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.490848 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.492252 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.492652 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.492654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.492654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.492719 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.492941 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.492942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.492942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.493093 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.493166 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.493389 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.493486 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:12:43.493752 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.482501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45011 (local address 127.0.0.1:59372)
0924 03:12:43.482685 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.482731 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.483412 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.483637 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.483641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.484045 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.484336 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.484341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.484975 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.484977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.486312 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.486316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.486677 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.486678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.486842 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.487332 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.487341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.490734 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.492327 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.492333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.492334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.492569 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.492786 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.492787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.492787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.492875 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.493158 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.493162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.493261 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.493365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.493508 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260924 03:12:43.494339 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:43.502364 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.494609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54244)
0924 03:12:43.494663 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.494672 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.494711 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.494996 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.494997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.495136 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.495173 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.495174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.495909 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.495910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.496348 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.496349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.496784 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.496785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.496846 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.497208 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.497212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.499661 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.501391 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.501392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.501392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.501544 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.501748 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.501749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.501749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.501811 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.502057 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.502057 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.502090 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.502139 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.502308 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:12:43.502507 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.494687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.61:46383)
0924 03:12:43.494842 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.494845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.494859 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:12:43.494874 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.494875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.494885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.494993 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.495246 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.495838 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.496434 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.496706 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.496785 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.499742 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.499748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.499748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.499754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.501317 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.501610 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.501613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.501613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.501662 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.501864 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.501864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.501864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.501998 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.502059 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.502159 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.502358 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260924 03:12:43.503057 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.503362 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.503268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39004)
0924 03:12:43.503309 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.503346 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:12:43.503492 23054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
I20260924 03:12:43.503914 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.503823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39016)
0924 03:12:43.503866 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.503889 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 03:12:43.504009 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35709: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:12:43.504665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskapYcIP/build/release/bin/kudu
/tmp/dist-test-taskapYcIP/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35709
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:12:43.550379 23028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } attempt: 3
I20260924 03:12:43.553220 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } attempt: 1
I20260924 03:12:43.553848 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.553731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39032)
0924 03:12:43.553787 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.553821 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:12:43.554392 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.554273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39042)
0924 03:12:43.554333 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.554364 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:12:43.554481 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35709: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:43.557790 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.550823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.0.98.61:46383)
0924 03:12:43.550886 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.550888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.550897 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:12:43.550921 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.550921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.550931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.550996 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.551342 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.551793 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.552822 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.553028 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.553078 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.555246 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.555247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.555247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.555255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.556697 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.557027 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.557029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.557029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.557082 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.557283 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.557283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.557284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.557390 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.557488 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.557639 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.557707 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:12:43.557797 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.550760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54258)
0924 03:12:43.550820 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.550832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.550875 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.551020 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.551021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.551192 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.551263 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.551265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.551880 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.551882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.552739 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.552741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.553096 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.553097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.553194 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.553637 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.553642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.555121 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.556792 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.556793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.556793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.556959 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.557157 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.557158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.557158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.557214 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.557484 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.557485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.557531 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.557616 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.557707 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:12:43.558127 23028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.558456 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.558364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39044)
0924 03:12:43.558413 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.558439 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 03:12:43.558758 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.558660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39050)
0924 03:12:43.558716 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.558738 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:12:43.558849 23028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35709: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
W20260924 03:12:43.582720 23110 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:12:43.582930 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:12:43.582968 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:12:43.583003 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:12:43.584097 23110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:12:43.584159 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:12:43.584183 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:12:43.584216 23110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:12:43.584246 23110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:12:43.585705 23110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45011,127.0.98.61:46383,127.0.98.60:35709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:12:43.585970 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:12:43.586237 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:12:43.589994 23028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } attempt: 1
I20260924 03:12:43.590423 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.590290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39060)
0924 03:12:43.590354 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.590395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:12:43.590797 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.590708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39070)
0924 03:12:43.590748 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.590769 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 03:12:43.590787 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:43.590798 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:12:43.591032 23028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35709: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35709: connect: Connection refused (error 111)
I20260924 03:12:43.591050 23110 server_base.cc:1034] running on GCE node
W20260924 03:12:43.590780 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:12:43.591295 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:12:43.591565 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:12:43.592717 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790219563592701 us; error 36 us; skew 500 ppm
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:12:43.595074 23110 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:12:43.595403 23110 webserver.cc:460] Webserver started at http://127.0.98.60:33423/ using document root <none> and password file <none>
I20260924 03:12:43.595561 23110 fs_manager.cc:359] Metadata directory not provided
I20260924 03:12:43.595597 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:12:43.595672 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:12:43.596536 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data/instance:
uuid: "2b3f3146748f44a88d569b64bddf0aff"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3b202b269646944fbeff80708817611b"
server_key_iv: "37dfd3b658015150ef767394d9d45dd4"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.596799 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal/instance:
uuid: "2b3f3146748f44a88d569b64bddf0aff"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3b202b269646944fbeff80708817611b"
server_key_iv: "37dfd3b658015150ef767394d9d45dd4"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.597921 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:12:43.598626 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:43.598774 23110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:12:43.598855 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal
uuid: "2b3f3146748f44a88d569b64bddf0aff"
format_stamp: "Formatted at 2026-09-24 03:12:43 on dist-test-slave-jtnz"
server_key: "3b202b269646944fbeff80708817611b"
server_key_iv: "37dfd3b658015150ef767394d9d45dd4"
server_key_version: "encryptionkey@0"
I20260924 03:12:43.598932 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:12:43.609962 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:12:43.611563 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:12:43.611660 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:12:43.615195 23110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35709
I20260924 03:12:43.615221 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35709 every 8 connection(s)
I20260924 03:12:43.615525 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:12:43.616420 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:12:43.617775 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.619207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskapYcIP/build/release/bin/kudu as pid 23110
I20260924 03:12:43.619284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskapYcIP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219310260848-392-0/minicluster-data/master-2/wal/instance
I20260924 03:12:43.619509 392 external_mini_cluster.cc:1428] Setting key 110a010cbc6cbe6594d5aa5aa23d4b31
I20260924 03:12:43.622614 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } attempt: 2
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:12:43.627748 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.618426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.98.62:45011)
0924 03:12:43.618493 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.618495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.618612 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:12:43.619372 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.619373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.619385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.619455 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.620247 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.620721 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.621697 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.621962 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.622025 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.624499 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.624500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.624500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.624507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.626170 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.626685 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.626686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.626686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.626765 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.627097 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.627098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.627098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.627222 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.627261 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.627570 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.627647 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:12:43.628111 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.618369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45011 (local address 127.0.0.1:59388)
0924 03:12:43.618503 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.618535 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.619219 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.619508 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.619511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.619854 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.620152 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.620158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.620812 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.620814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.621614 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.621616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.622068 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.622069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.622239 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.622814 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.622822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.624412 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.626291 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.626297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.626298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.626601 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.626849 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.626850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.626850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.627007 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.627303 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.627307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.627433 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.627827 (+ 394us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.627968 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":75,"threads_started":1}
I20260924 03:12:43.628545 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.629227 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.619879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.98.60:35709)
0924 03:12:43.620048 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.620051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.620068 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:12:43.620137 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.620141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.620157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.620253 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.620954 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.621782 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.622779 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.623207 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.623290 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.625821 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.625828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.625828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.625835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.627686 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.628263 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.628265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.628266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.628312 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.628475 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.628476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.628477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.628606 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.628884 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.629058 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.629128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 03:12:43.629709 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.622857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39086)
0924 03:12:43.622913 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.622922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.622953 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.623156 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.623158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.623310 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.623358 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.623360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.624286 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.624287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.624962 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.624964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.625297 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.625298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.625358 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.625658 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.625674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.626921 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.628757 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.628758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.628758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.628944 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.629123 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.629124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.629124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.629175 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.629431 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.629432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.629476 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.629533 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.629620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260924 03:12:43.629993 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.622895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.0.98.60:35709)
0924 03:12:43.623038 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.623039 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.623046 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:43.623061 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.623062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.623070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.623129 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.623448 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.624194 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.625038 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.625223 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.625296 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.627052 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.627053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.627053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.627059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.628680 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.629010 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.629010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.629011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.629059 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.629234 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.629235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.629235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.629349 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.629410 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.629821 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.629884 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260924 03:12:43.631263 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: Bootstrap starting.
I20260924 03:12:43.632171 23103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:43.632485 23103 log.cc:824] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:43.633188 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: No bootstrap required, opened a new log
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:12:43.635272 23103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.635430 23103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a7af97fb0a4a48abb6b1bced28e788, State: Initialized, Role: FOLLOWER
I20260924 03:12:43.635529 23103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.635823 23103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:43.635949 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.628779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.0.98.61:46383)
0924 03:12:43.628815 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.628817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.628848 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:12:43.628896 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.628897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.628905 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.628952 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.629468 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.630024 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.630937 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.631179 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.631235 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.632952 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.632953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.632954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.632960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.634364 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.634707 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.634708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.634709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.634756 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.634983 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.634984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.634984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.635127 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.635201 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.635750 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.635825 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260924 03:12:43.636200 23185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.636346 23185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.636740 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.628725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46383 (local address 127.0.0.1:54262)
0924 03:12:43.628814 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.628822 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.628857 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.628954 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.628954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.629080 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.629365 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.629366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.630253 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.630254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.630859 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.630861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.631252 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.631253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.631316 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.631643 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.631648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.632846 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.634456 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.634457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.634457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.634632 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.634838 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.634838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.634839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.634897 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.635206 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.635207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.635292 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.635863 (+ 571us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.636681 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:12:43.637018 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:12:43.637639 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.630216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.0.98.62:45011)
0924 03:12:43.630247 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.630249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.630256 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:43.630274 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.630274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.630281 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.630321 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.630741 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.631274 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.631950 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.632185 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.632244 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.634925 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.634926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.634927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.634932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.636553 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.636931 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.636932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.636932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.636987 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.637207 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.637208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.637209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.637322 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.637464 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.637496 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.637559 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:12:43.637802 23196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:43.637856 23196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:43.637948 23035 master_runner.cc:186] Error getting master registration for 127.0.98.62:45011: OK, instance_id { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" instance_seqno: 1790219563325198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:43.640537 23035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:12:43.644186 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.637204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39098)
0924 03:12:43.637232 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.637240 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.637271 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.637917 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.637918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.638057 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.638083 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.638085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.638846 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.638846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.639517 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.639518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.639907 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.639908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.639945 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.640283 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.640288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.641534 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.643178 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.643179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.643179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.643340 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.643502 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.643503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.643503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.643574 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.643879 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.643880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.643924 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.644000 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.644087 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 03:12:43.644243 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.637292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.60:35709)
0924 03:12:43.637332 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.637333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.637340 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:43.637354 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.637355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.637362 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.637408 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.638153 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.638762 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.639588 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.639822 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.639873 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.641616 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.641617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.641618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.641623 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.643103 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.643409 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.643410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.643410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.643448 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.643627 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.643628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.643628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.643779 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.643842 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.644020 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.644135 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":29}
I20260924 03:12:43.645381 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: Bootstrap starting.
I20260924 03:12:43.645800 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.638331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.0.98.61:46383)
0924 03:12:43.638358 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.638360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.638365 (+ 5us) server_negotiation.cc:407] Connection header received
0924 03:12:43.638381 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.638382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.638389 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.638437 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.638959 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.639518 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.640318 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.640581 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.640643 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.643060 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.643061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.643061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.643067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.644561 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.644935 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.644936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.644936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.644983 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.645140 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.645140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.645140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.645236 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.645331 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.645668 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.645727 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:12:43.645908 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:43.646150 23178 log.cc:824] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:43.646848 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: No bootstrap required, opened a new log
I20260924 03:12:43.646953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:12:43.647753 23002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9e6fa5c9ae324f2e914f6adfbc724280"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59402
I20260924 03:12:43.648605 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.648774 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3f3146748f44a88d569b64bddf0aff, State: Initialized, Role: FOLLOWER
I20260924 03:12:43.648860 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.649103 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:43.649073 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.649161 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:12:43.650971 23210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:43.651026 23210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:43.651005 23110 master_runner.cc:186] Error getting master registration for 127.0.98.62:45011: OK, instance_id { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" instance_seqno: 1790219563325198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:12:43.653040 23110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:12:43.666265 23028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } attempt: 2
Sep 24 03:12:43 dist-test-slave-jtnz krb5kdc[22947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:12:43.673437 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.666509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35709 (local address 127.0.0.1:39106)
0924 03:12:43.666552 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 03:12:43.666562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:12:43.666601 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:12:43.666742 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:12:43.666743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:12:43.666909 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:12:43.666946 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.666947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.667785 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.667786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.668494 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:12:43.668496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.668905 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.668906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:12:43.668953 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.669350 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:12:43.669354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:12:43.670700 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:12:43.672415 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.672416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.672417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.672631 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.672821 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:12:43.672822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:12:43.672822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:12:43.672884 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:12:43.673130 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:12:43.673131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:12:43.673173 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:12:43.673242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:12:43.673336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 03:12:43.673439 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.666591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.98.60:35709)
0924 03:12:43.666639 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.666641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.666648 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:12:43.666666 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.666666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.666675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.666726 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.667007 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.667704 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.668580 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.668843 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.668909 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.670779 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.670780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.670780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.670786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.672325 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.672698 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.672699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.672700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.672753 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.672949 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.672949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.672950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.673063 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.673125 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.673290 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.673359 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:12:43.675206 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: Bootstrap starting.
I20260924 03:12:43.675825 23028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: Neither blocks nor log segments found. Creating new log.
I20260924 03:12:43.676105 23028 log.cc:824] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: Log is configured to *not* fsync() on all Append() calls
I20260924 03:12:43.676884 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: No bootstrap required, opened a new log
I20260924 03:12:43.678138 23028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.678288 23028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6fa5c9ae324f2e914f6adfbc724280, State: Initialized, Role: FOLLOWER
I20260924 03:12:43.678365 23028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.678637 23028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:12:43.678696 23002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:12:43.678761 23002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:43.679454 23002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.679894 23077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48a7af97fb0a4a48abb6b1bced28e788"
I20260924 03:12:43.680013 23077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:12:43.680924 23077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6fa5c9ae324f2e914f6adfbc724280 in term 1.
I20260924 03:12:43.681217 22976 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a7af97fb0a4a48abb6b1bced28e788, 9e6fa5c9ae324f2e914f6adfbc724280; no voters:
I20260924 03:12:43.681226 23002 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [CANDIDATE]: Term 1 election: Requested vote from peers 48a7af97fb0a4a48abb6b1bced28e788 (127.0.98.61:46383), 2b3f3146748f44a88d569b64bddf0aff (127.0.98.60:35709)
I20260924 03:12:43.682552 23152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2b3f3146748f44a88d569b64bddf0aff"
I20260924 03:12:43.682654 23152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [term 0 FOLLOWER]: Advancing to term 1
W20260924 03:12:43.683316 23224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:12:43.683418 23224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:12:43.683439 23152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6fa5c9ae324f2e914f6adfbc724280 in term 1.
I20260924 03:12:43.683758 22960 master_runner.cc:263] No leader master found from master 9e6fa5c9ae324f2e914f6adfbc724280
I20260924 03:12:43.683797 22960 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:12:43.684351 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.684435 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.684485 23225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:12:43.684532 23225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [term 1 LEADER]: Becoming Leader. State: Replica: 9e6fa5c9ae324f2e914f6adfbc724280, State: Running, Role: LEADER
I20260924 03:12:43.684602 23225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } }
I20260924 03:12:43.685338 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6fa5c9ae324f2e914f6adfbc724280. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.685405 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:43.688841 23235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:12:43.689013 23235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:12:43.692615 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.683925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.0.98.60:35709)
0924 03:12:43.683959 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.683960 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.684095 (+ 135us) server_negotiation.cc:407] Connection header received
0924 03:12:43.684192 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.684192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.684201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.684228 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.685215 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.685853 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.686994 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.687280 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.687318 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.689130 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.689131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.689131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.689138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.690767 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.691529 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.691531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.691531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.691586 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.692054 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.692055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.692055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.692176 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.692267 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.692467 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.692534 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:43.694149 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.684136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.62:45011)
0924 03:12:43.684170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.684172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.685578 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 03:12:43.685661 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.685661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.685669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.685694 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.686408 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.687120 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.688105 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.688381 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.688416 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.691029 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.691030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.691031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.691037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.692491 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.693340 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.693341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.693342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.693390 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.693606 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.693607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.693607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.693730 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.693791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.693993 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.694063 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:12:43.695488 23151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [term 1 FOLLOWER]: Refusing update from remote peer 9e6fa5c9ae324f2e914f6adfbc724280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:43.695746 23225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:43.696868 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6fa5c9ae324f2e914f6adfbc724280. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.696964 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.697160 23077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [term 1 FOLLOWER]: Refusing update from remote peer 9e6fa5c9ae324f2e914f6adfbc724280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:12:43.697387 23225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:12:43.698453 23185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6fa5c9ae324f2e914f6adfbc724280. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.698529 23185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.699501 23199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.699592 23199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.699803 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.699887 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:43.699993 23236 mvcc.cc:205] Tried to move back new op lower bound from 7332739332894539776 to 7332739332854317056. Current Snapshot: MvccSnapshot[applied={T|T < 7332739332894539776}]
I20260924 03:12:43.700428 23237 mvcc.cc:205] Tried to move back new op lower bound from 7332739332894539776 to 7332739332854317056. Current Snapshot: MvccSnapshot[applied={T|T < 7332739332894539776}]
I20260924 03:12:43.700634 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:43.684964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.0.98.61:46383)
0924 03:12:43.684996 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.684997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.690637 (+ 5640us) server_negotiation.cc:407] Connection header received
0924 03:12:43.690722 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.690723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.690731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.690760 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.693605 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.694335 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.695472 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:43.695757 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:43.695793 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:43.697417 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:12:43.697418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:12:43.697419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:12:43.697425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:12:43.699002 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.699673 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.699674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.699674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.699761 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:12:43.700077 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:12:43.700078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:12:43.700078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:12:43.700193 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:12:43.700252 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:43.700457 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:12:43.700551 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:12:43.702533 23235 catalog_manager.cc:1347] Generated new cluster ID: b45b33f3ffc64f169e93987d5b53467e
I20260924 03:12:43.702582 23235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:12:43.711071 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.711148 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280 [sys.catalog]: This master's current role is: LEADER
I20260924 03:12:43.711527 23185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6fa5c9ae324f2e914f6adfbc724280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45011 } } peers { permanent_uuid: "48a7af97fb0a4a48abb6b1bced28e788" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46383 } } peers { permanent_uuid: "2b3f3146748f44a88d569b64bddf0aff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35709 } } }
I20260924 03:12:43.711591 23185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:12:43.712360 23239 mvcc.cc:205] Tried to move back new op lower bound from 7332739332894539776 to 7332739332854317056. Current Snapshot: MvccSnapshot[applied={T|T < 7332739332894539776 or (T in {7332739332894539776})}]
I20260924 03:12:43.712726 23235 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:12:43.714457 23235 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:12:43.727617 23235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e6fa5c9ae324f2e914f6adfbc724280: Generated new TSK 0
I20260924 03:12:43.727756 23235 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:12:43.754873 23183 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.753539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.0.98.60:35709)
0924 03:12:43.753576 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.753578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.754212 (+ 634us) server_negotiation.cc:407] Connection header received
0924 03:12:43.754323 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.754324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.754334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.754365 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.754812 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39124: BlockingRecv error: recv got EOF from 127.0.0.1:39124 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:12:43.755581 23031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.753919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.0.98.62:45011)
0924 03:12:43.753957 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.753959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.755112 (+ 1153us) server_negotiation.cc:407] Connection header received
0924 03:12:43.755196 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.755197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.755207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.755234 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.755521 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59420: BlockingRecv error: recv got EOF from 127.0.0.1:59420 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:12:43.756489 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.755870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.98.61:46383)
0924 03:12:43.755908 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.755910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.756022 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:12:43.756102 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.756103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.756111 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.756136 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.756439 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54272: BlockingRecv error: recv got EOF from 127.0.0.1:54272 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 03:12:43.758354 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.757739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.0.98.61:46383)
0924 03:12:43.757767 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.757769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.757911 (+ 142us) server_negotiation.cc:407] Connection header received
0924 03:12:43.758001 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.758002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.758015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.758039 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.758322 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54280: BlockingRecv error: recv got EOF from 127.0.0.1:54280 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260924 03:12:43.761097 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.760407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.0.98.61:46383)
0924 03:12:43.760435 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.760437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.760617 (+ 180us) server_negotiation.cc:407] Connection header received
0924 03:12:43.760700 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.760701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.760709 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.760734 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.761061 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54284: BlockingRecv error: recv got EOF from 127.0.0.1:54284 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:12:43.764798 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.764105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.0.98.61:46383)
0924 03:12:43.764138 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.764139 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.764325 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:12:43.764433 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.764434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.764443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.764471 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.764762 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54290: BlockingRecv error: recv got EOF from 127.0.0.1:54290 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:12:43.770437 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.769746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.0.98.61:46383)
0924 03:12:43.769773 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.769774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.769944 (+ 170us) server_negotiation.cc:407] Connection header received
0924 03:12:43.770022 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.770023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.770032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.770055 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.770403 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54298: BlockingRecv error: recv got EOF from 127.0.0.1:54298 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 03:12:43.780082 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.779358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.0.98.61:46383)
0924 03:12:43.779398 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.779399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.779684 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:12:43.779762 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.779763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.779771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.779795 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.780048 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54308: BlockingRecv error: recv got EOF from 127.0.0.1:54308 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 03:12:43.797900 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.797201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.0.98.61:46383)
0924 03:12:43.797237 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.797240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.797446 (+ 206us) server_negotiation.cc:407] Connection header received
0924 03:12:43.797530 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.797531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.797540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.797565 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.797851 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54324: BlockingRecv error: recv got EOF from 127.0.0.1:54324 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:12:43.831804 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.831144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.0.98.61:46383)
0924 03:12:43.831209 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.831211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.831310 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:12:43.831382 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.831383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.831393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.831460 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.831754 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54332: BlockingRecv error: recv got EOF from 127.0.0.1:54332 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 03:12:43.897656 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:43.897034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.98.61:46383)
0924 03:12:43.897091 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:43.897093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:43.897164 (+ 71us) server_negotiation.cc:407] Connection header received
0924 03:12:43.897221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:43.897222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:43.897237 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:43.897324 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:43.897587 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54334: BlockingRecv error: recv got EOF from 127.0.0.1:54334 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 03:12:44.027459 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:44.026756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.0.98.61:46383)
0924 03:12:44.026825 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.026827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.026936 (+ 109us) server_negotiation.cc:407] Connection header received
0924 03:12:44.027021 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.027022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.027034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.027105 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:44.027412 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54338: BlockingRecv error: recv got EOF from 127.0.0.1:54338 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 03:12:44.285429 23108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:12:44.284618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.0.98.61:46383)
0924 03:12:44.284683 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.284685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.284739 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:12:44.284784 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.284785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.284795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.284867 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:12:44.285365 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54348: BlockingRecv error: recv got EOF from 127.0.0.1:54348 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:12:44.638273 23196 catalog_manager.cc:1259] Loaded cluster ID: b45b33f3ffc64f169e93987d5b53467e
I20260924 03:12:44.638334 23196 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: loading cluster ID for follower catalog manager: success
I20260924 03:12:44.639159 23196 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: acquiring CA information for follower catalog manager: success
I20260924 03:12:44.639470 23196 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48a7af97fb0a4a48abb6b1bced28e788: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:44.651331 23210 catalog_manager.cc:1259] Loaded cluster ID: b45b33f3ffc64f169e93987d5b53467e
I20260924 03:12:44.651376 23210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: loading cluster ID for follower catalog manager: success
I20260924 03:12:44.652040 23210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: acquiring CA information for follower catalog manager: success
I20260924 03:12:44.652290 23210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b3f3146748f44a88d569b64bddf0aff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:12:44.801004 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.798473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.0.98.61:46383)
0924 03:12:44.798588 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.798590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.798653 (+ 63us) server_negotiation.cc:407] Connection header received
0924 03:12:44.798745 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.798746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.798805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.798885 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:44.799031 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.799519 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.800215 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.800408 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.800475 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:44.800634 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:44.800747 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:44.800856 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:44.800878 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 03:12:44.800907 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
I20260924 03:12:44.804658 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.802659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.60:35709)
0924 03:12:44.802758 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.802760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.802791 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:12:44.802832 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.802833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.802865 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.802922 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:44.802983 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.803459 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.804087 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.804262 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.804307 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:44.804418 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:44.804503 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:44.804554 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:44.804588 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 03:12:44.804603 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":57,"threads_started":1}
I20260924 03:12:44.808681 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.806170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.98.60:35709)
0924 03:12:44.806232 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.806234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.806338 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:12:44.806406 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.806407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.806415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.806460 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:44.806717 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.807333 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808009 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.808250 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808307 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:44.808438 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:44.808466 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:44.808493 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:44.808600 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:12:44.808618 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:12:44.809130 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.806065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.98.62:45011)
0924 03:12:44.806254 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.806256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.806367 (+ 111us) server_negotiation.cc:407] Connection header received
0924 03:12:44.806412 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.806413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.806467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.806558 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:44.806680 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.807134 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808062 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.808345 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808420 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:44.808612 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:44.808704 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:44.808890 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:44.809004 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 03:12:44.809048 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
I20260924 03:12:44.809167 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:12:44.806137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.0.98.61:46383)
0924 03:12:44.806173 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:12:44.806200 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 03:12:44.806404 (+ 204us) server_negotiation.cc:407] Connection header received
0924 03:12:44.806470 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:12:44.806471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:12:44.806479 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:12:44.806527 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:12:44.806629 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.807305 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808210 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:12:44.808483 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:12:44.808542 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:12:44.808762 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:12:44.808791 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:12:44.808842 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:12:44.809058 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:12:44.809082 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:12:44.810680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 22960
I20260924 03:12:44.815655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 23035
I20260924 03:12:44.820225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskapYcIP/build/release/bin/kudu with pid 23110
2026-09-24T03:12:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2717 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5392 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (254565 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST